Skip to main content

         This documentation site is for previous versions. Visit our new documentation site for current releases.      

Changing the default number of partitions per topic

Updated on July 5, 2022

To achieve high data consumption throughput and concurrency, the Stream service uses the Kafka concept of partitions. Starting in Pega Platform version 8.6.4, the default number of partitions per topic is six, which means that up to six clients can simultaneously consume data from a Stream data set.

Note: If the number of BackgroundProcessing or RealTime nodes is higher than six, then only six nodes are occupied. If more than one consumption thread per node is configured, then the number of consumers is the number of nodes multiplied by the number of threads.

In most of the Pega Platform versions it is possible to customize the default number of partitions. For more information, see The support for default partition number customization.

In Pega Platform versions earlier than 8.6.4, the default number of partitions was set to 20. After a system update to version 8.7, the number of partitions of existing Stream data sets does not change. However, if you truncate the pr_data_stream_nodes table, the number of partitions reverts to the new default of six.

In most cases, six partitions is enough for effective processing. However, if your application needs a higher consumption throughput and thus requires more than six processing nodes, you can increase the number of stream partitions. Ensure that the number of partitions per broker does not exceed the limit.

To change the global default number of partitions in the cluster, create and configure the following dynamic system setting:

  1. In the header of Dev Studio, click CreateSysAdminDynamic System Settings.
  2. On the New tab, configure the following parameters:
    1. In the Dynamic system settings short description field, enter the purpose of this setting.
      For example: You can use the following description: Global default number of partitions per topic
    2. In the Owning Ruleset field, enter: Pega-Engine
    3. In the Setting Purpose field, enter: prconfig/dsm/services/stream/pyTopicPartitionsCount/default
  3. Click Create and open.
  4. In the Value field, enter the new global default partitions count per topic.
  5. Click Save.
  6. Restart the server.
Result: The new setting is applied only to newly created partitions.

Have a question? Get answers now.

Visit the Support Center to ask questions, engage in discussions, share ideas, and help others.

Did you find this content helpful?

Want to help us improve this content?

We'd prefer it if you saw us at our best. is not optimized for Internet Explorer. For the optimal experience, please use:

Close Deprecation Notice
Contact us