Skip to main content


         This documentation site is for previous versions. Visit our new documentation site for current releases.      
 

Operating the Stream service

Updated on July 5, 2022

Learn how to do a rolling and full restart of your Stream service, to manually scale up and scale down the service, and to recover the service after a failure.

Note:

Starting in version 8.7, the use of an internal Kafka is deprecated. On-premises systems that have been updated from earlier versions of Pega Platform can continue to use Kafka in embedded mode. However, to ensure future compatibility, do not create any new environments using embedded Kafka. When configuring the Stream service in a new environment, use external Kafka.

When the Stream service is running in External Mode, there are no Stream Tier instances and none of the procedures mentioned below are required.

Rolling restart Stream Tier

The Stream tier can be "rehydrated" by restarting one instance at a time. Terminate each instance in a graceful fashion. It may take up to 10 minutes to shut down a Stream instance as the data ownership has to be handed over to the remaining nodes. Do not forcefully terminate instances during this period.

When an instance comes back online, wait for a grace period of 5 minutes before you restart the next instance. During the grace period, the Stream node replicates the data which might have been missed during the period that the instance was unreachable.

Full restart Stream Tier

In case a rolling restart is not possible for some reason, you can do a full restart.

Take the nodes out of service one instance at a time. Each instance has to be terminated in a graceful fashion. It make take up to 10 minutes to shut down a Stream instance as the data ownership has to be handed over to the remaining nodes. Do not forcefully terminate instances during this period.

Bring instances back up in reverse order.

Scaling up Stream Tier

Stream service does not support automatic scaling, however, a manual scale-up is possible.

  1. In the header of Dev Studio, click ConfigureDecisioning InfrastructureServicesStream.
  2. Ensure that all Stream instances are up and have the status NORMAL.
  3. Ensure that under-replicated and offline partitions are 0 for all instances in the list.
  4. If under-replicated and offline partitions are greater than 0, wait until they get to 0.
    Scaling up the Stream service
    Stream landing page in Dev Studio
  5. Start an additional Stream instance.
    Note: You can add only one instance at a time. Repeat the procedure if you need to add more instances.

Scaling down Stream Tier

Stream service does not support automatic scaling, however a manual scale-down is possible:

  1. In the header of Dev Studio, click ConfigureDecisioning InfrastructureServicesStream.
  2. Select a node you would like to take out of the service, and click ExecuteDecommission.
  3. Wait until the instance is removed from the list.
  4. Terminate the node.

Full recovery Stream Tier

In case of a catastrophic failure, the following procedure can be followed to recover the Stream tier:

Important: This procedure results in Kafka Data Loss.
  1. Bring all Stream service instances down.
  2. In the header of Dev Studio, click ConfigureDecisioning InfrastructureServicesStream.
  3. Decommission all Stream service instances.
  4. Truncate the Stream service tables in the Pega Platform data schema:
    • truncate table pr_data_stream_nodes
    • truncate table pr_data_stream_sessions
    • truncate table pr_data_stream_node_updates
  5. Delete Kafka-data directory from the Stream instances.
    Note: This directory, by default is created under Tomcat folder.
  6. Bring the Stream instances back up one instance at a time.

Have a question? Get answers now.

Visit the Support Center to ask questions, engage in discussions, share ideas, and help others.

Did you find this content helpful?

Want to help us improve this content?

We'd prefer it if you saw us at our best.

Pega.com is not optimized for Internet Explorer. For the optimal experience, please use:

Close Deprecation Notice
Contact us