Skip to main content


         This documentation site is for previous versions. Visit our new documentation site for current releases.      
 

Sizing queue processors and Stream data sets individually by changing the number of partitions of a topic

Updated on July 5, 2022

Each queue processor in Pega Platform is associated with a Stream data set in the Stream service. You can size individual queue processors and Stream data sets by changing the number of partitions of a topic associated with a particular queue processor or Stream data set.

For example, if you expect a potential increase in traffic for some queue processors during a product launch or the holiday season, you can increase the number of partitions for these queue processors.

  1. In Dev Studio, search for the pxAlterStreamPartitions activity.
  2. Click ActionsRun.
  3. In the Run Activity: AlterStreamPartitions dialog box, in the stream field, enter one of the following values:
    • If the topic is associated with a queue processor, enter the name of the queue processor in uppercase letters.
    • If the topic is associated with a Stream data set, enter the name of the Stream data set.
  4. In the partitions field, enter the desired number of partitions.
  5. Depending on the number of partitions, perform one of the following actions:
    • If you are decreasing the number of partitions, select the allowDataLoss checkbox.
    • If you are increasing the number of partitions, leave the allowDataLoss checkbox cleared.

    When you decrease the number of partitions, the system deletes the existing topic, and then recreates the topic with the new partition count. Existing data is lost. You need to select the allowDataLoss checkbox to confirm that data loss is allowed. When you increase the number of partitions, existing data is not impacted.

    Run Activity: AlterStreamPartitions dialog box
    The number of partitions for a RealTimeInteractions data set is set to five. The allowDataloss checkbox is selected.
  6. In the upper-right corner of the Run Activity: AlterStreamPartitions dialog box, click Run.
    Result: The real-time data flow that is associated with the queue processor automatically detects the change in the number of partitions in the source data set and adjusts the details of the run.
    Partitions count in a data flow run
    The number of partitions for the Write Data To Interaction Files is six.

    For more information about partition metrics for a data flow run, see Managing data flow runs.

  • Previous topic The support for default partition number customization
  • Next topic Best practices for Stream service configuration

Have a question? Get answers now.

Visit the Support Center to ask questions, engage in discussions, share ideas, and help others.

Did you find this content helpful?

Want to help us improve this content?

We'd prefer it if you saw us at our best.

Pega.com is not optimized for Internet Explorer. For the optimal experience, please use:

Close Deprecation Notice
Contact us