Uploaded image for project: 'CFA MX '
  1. CFA MX
  2. CFAMX-2933

Evenly spread processing Queue Messages over a period time

    Details

    • Sprint:
      CFAMX 2019R2 Sprint 5, CFAMX 2019R2 Hardening
    • SCRUM Team:
      Brotherhood of Mutants
    • Story Points:
      5
    • Work Type Classification:
      Sustaining

      Description

      When retrieving forecasts from CFA, we try to process the forecasts as fast as we can. This results in a major hit to the CPU on the app server for a short amount of time. We'd prefer to spread the processing of these messages out over a longer period of time so that the cpu isn't slammed at any specific time.

      Create a reusable solution to pace the Forecast messages, in case we ever want to do this for a future message type:

      1) Create a table, "tbQueueProcessingDailyPacing" (or similar name) to hold the pacing configuration for a queue. It should have these columns:

      • QueueName - name of the queue to pace
      • DailyDeadline - time all messages must be finished by
      • PacingThreshold - minimum number of messages in the queue before we bother pacing
      • PollingDelay - how frequently a queue should recalculate the pace needed to finish all messages by the deadline

      2) Add a "GetQueueName() as String" function to ITaskController, and an implementation in TaskPollingQueue (see commit)

      3) Add a DelayProcessing() method to RetrieveForecastMessage. Call this method at the start of the PerformAction() function to delay the action as long as needed. (see commit)

      4) Update the Task Message so that someone looking at the queue in Server Manager can tell that it's processing is being delayed/paced.

      5) Documentation - Create a db diagram for the new table, a flow diagram for the changes to the Forecast workflow, and a Confluence article about the new configuration via the table

       

      Acceptance Criteria:

      The queues configured in the new table will process messages at their configured pace, rather than as fast as possible like today.

      Documentation of new process and configuration:

      Database Diagram: https://drive.google.com/open?id=1WRvDefznWxQXzQncaOLed3F3c2g9R_rL

      Architecture Flow Diagram: https://drive.google.com/open?id=1pdwdqRQ4vSDYfWlFxe3sTeDFUG_Ju_Bz

      Confluence Article: https://cfacorp.atlassian.net/wiki/spaces/IN/pages/102073106/Server+Manager+Queue+Pacing

        Attachments

          Issue Links

          There are no Sub-Tasks for this issue.

            Activity

              People

              • Assignee:
                soma.dixon Soma Dixon (Inactive)
                Reporter:
                david.nayyar David Nayyar (Inactive)
              • Watchers:
                1 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - 38h 40m Original Estimate - 38h 40m
                  38h 40m
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 51h 47m
                  51h 47m

                    PagerDuty

                    Error rendering 'com.pagerduty.jira-server-plugin:PagerDuty'. Please contact your Jira administrators.