Rolling restart Stream Tier

The Stream tier can be "rehydrated" by restarting one instance at a time. Terminate each instance in a graceful fashion. It may take up to 10 minutes to shut down a Stream instance as the data ownership has to be handed over to the remaining nodes. Do not forcefully terminate instances during this period.

When an instance comes back online, wait for a grace period of 5 minutes before you restart the next instance. During the grace period, the Stream node replicates the data which might have been missed during the period that the instance was unreachable.

Full restart Stream Tier

In case a rolling restart is not possible for some reason, you can do a full restart.

Take the nodes out of service one instance at a time. Each instance has to be terminated in a graceful fashion. It make take up to 10 minutes to shut down a Stream instance as the data ownership has to be handed over to the remaining nodes. Do not forcefully terminate instances during this period.

Bring instances back up in reverse order.

Scaling up Stream Tier

Stream service does not support automatic scaling, however, a manual scale-up is possible.

1. In the header of Dev Studio, click ConfigureDecisioning InfrastructureServicesStream.
2. Ensure that all Stream instances are up and have the status NORMAL.
3. Ensure that under-replicated and offline partitions are 0 for all instances in the list.
4. If under-replicated and offline partitions are greater than 0, wait until they get to 0.

   

5. Start an additional Stream instance.Note: You can add only one instance at a time. Repeat the procedure if you need to add more instances.

Scaling down Stream Tier

Stream service does not support automatic scaling, however a manual scale-down is possible:

1. In the header of Dev Studio, click ConfigureDecisioning InfrastructureServicesStream.
2. Select a node you would like to take out of the service, and click ExecuteDecommission.
3. Wait until the instance is removed from the list.
4. Terminate the node.

Full recovery Stream Tier

In case of a catastrophic failure, the following procedure can be followed to recover the Stream tier:Important: This procedure results in Kafka Data Loss.

1. Bring all Stream service instances down.
2. In the header of Dev Studio, click ConfigureDecisioning InfrastructureServicesStream.
3. Decommission all Stream service instances.
4. Truncate the Stream service tables in the Pega Platform data schema:
   • truncate table pr_data_stream_nodes
   • truncate table pr_data_stream_sessions
   • truncate table pr_data_stream_node_updates
5. Delete Kafka-data directory from the Stream instances. Note: This directory, by default is created under Tomcat folder.
6. Bring the Stream instances back up one instance at a time.