Skip to main content

         This documentation site is for previous versions. Visit our new documentation site for current releases.      

Data flow run priorities

Updated on July 5, 2022

Use the data flow service to protect the system against running out of resources. The data flow service can automatically queue and execute data flow runs according to configured priorities that indicate the relative importance of each run. For example, you can set the priority of an important data flow run to High to ensure that it will be performed first.

To protect the system from running out of resources due to a large number of data flows that run simultaneously, you can indicate the maximum number of data flow runs for a given node type, as well as change the maximum active runs threshold at run time. For more information, see Data flow run limits.

By setting data flow priorities, you can specify the importance of each data flow in a given data flow service type. Data flows are executed automatically, according to two factors:

  • The relative importance (priority) of the run that you can set to Low, Medium, or High.
  • The order of submission (activation time) of the run.

The runs that cannot execute when triggered by the user or system event (for example, low priority runs or runs that exceed the set data flow service limit) are queued automatically. The data flow service attempts to schedule the queued runs of equal priority in the order of submission, from the earliest to the latest. When the resources become available, then the oldest highest priority run starts. If the run is completed, stopped, or fails, then the current oldest run with the highest priority moves to the front of the queue. If the data flow service can perform a run with a higher priority, then the lower priority runs will remain queued.

For example:

Consider a scenario where the data flow service limit is set to five runs and all runs are currently in progress.

Data flow run queue
Diagram of data flow service run showing five in progress runs with different priority levels and activation time.
  1. The data flow service analyzes the run priority first, and then analyzes its activation time. When the limit of five active runs is reached, a new run (DF-6) with a medium priority level starts instead of the DF-2 run which has the lowest priority and has been added most recently. Another newly added run (DF-7) has a low priority level, therefore the run is queued.
    New data flow runs are added to the queue
    Diagram of data flow service run showing two new runs added to the previous queue.
  2. If you add another run with a higher priority (DF-8), that run starts first, while the DF-6 run moves to the queue.
    Automatic prioritization of a high priority run in the queue
    Diagram of data flow service run showing a high priority run being performed first.
  3. When the highest priority active run finishes (DF-3), the data flow service automatically starts the next highest priority, earliest run from the queue (DF-6).
    Completion of the previous runs allows previously queued runs to proceed
    Diagram of data flow service run showing how the completion of a high priority run allows previously queued lower priority runs to proceed.
  • Prioritizing data flow runs

    Secure the most optimal use of system resources by indicating the relative importance (priority) of each data run. The system automatically performs or queues data flow runs according to the set priorities and activation times, limited by the maximum number of active runs.

Have a question? Get answers now.

Visit the Support Center to ask questions, engage in discussions, share ideas, and help others.

Did you find this content helpful?

Want to help us improve this content?

We'd prefer it if you saw us at our best. is not optimized for Internet Explorer. For the optimal experience, please use:

Close Deprecation Notice
Contact us