Changes to the architecture of the Data Flow service
Valid from Pega Version 8.4
In Pega Platform™ 8.4, the architecture of batch and real-time data flows uses improved node handling to increase the stability of data flow runs. As a result, there are fewer interactions with the database and between the nodes, resulting in increased resilience of the Data Flow service.
If you upgrade from a previous version of Pega Plaftorm, see the following list for an overview of the changes in the behavior of the Data Flow service compared to previous versions:
- Responsiveness
- Updates to lifecycle actions
- Starting a run
- Triggering pre- and post-activities
- Selecting a node fail policy
- No service nodes and active runs
Responsiveness
Nodes no longer communicate and trigger each other, but run periodic tasks instead. As such, triggering a new run does not cause the service nodes to immediately start the run. Instead, the run starts a few seconds later. The same applies to user actions such as stopping, starting, and updating the run. The system also processes topology changes as periodic tasks, so it might take a few minutes for new nodes to join runs, or for partitions to redistribute when a node leaves a run.
Updates to lifecycle actions
To make lifecycle actions more intuitive, the Stop action consolidates both the Stop and Pause actions. The Start action consolidates both the Resume and Start actions.
You can resume or restart stopped and failed runs with the Start and Restart actions. The Start action is only available for resumable runs and continues the run from where it stopped. The Restart action causes the run to process from the beginning. Completed runs can only be restarted. If a run completes with failures, you can restart it from the beginning, or process only the errors by using the Reprocess failures action.
Starting a run
New data flow runs have the Initializing status, and start automatically. You no longer need to manually start a new run, so the New status is now removed.
If there are no nodes available to process a run, the run gets the Queued status and waits for an available node.
Triggering pre- and post-activities
The system now triggers pre-activities on a random service node, rather than on the node that triggered the run.
The system triggers post-activities only for runs that complete, fail, or complete with failures. If you manually stop a run with the Stop action, the post-activity does not trigger. However, restarting the run with the Restart action triggers first the post-activity, and then the pre-activity.
You can no longer choose to run pre- and post-activities on all nodes.
Selecting a node fail policy
For resumable runs, you can no longer select a node fail policy. If a node fails, the partitions assigned to that node automatically continue the run on different nodes.
For non-resumable runs, you can choose to restart the partitions assigned to the failed node on different nodes, or to fail the partitions assigned to the failed node.
No service nodes and active runs
If the last data flow node for an in-progress run fails, the run remains in the In Progress state, even if no processing takes place. This behavior results from the fact that data flow architecture now prevents unrelated nodes from affecting runs.
Text predictions simplify the configuration of text analytics for conversational channels
Valid from Pega Version 8.6
Enable text analytics for your conversational channels, such as email and chatbot, by configuring text predictions that manage the text models for your channels. This new type of prediction in Prediction Studio consolidates the AI for analyzing the messages in your conversational channels in one place and replaces the text analyzer rule in Dev Studio.
Through text predictions, you can efficiently configure the outcomes that you want to predict by analyzing the text in your channels:
- Topics (ticket booking, subscription cancellation, support request)
- Sentiments (positive, neutral, negative)
- Entities (people, organizations, airport codes)
- Languages
You can train and build the models that predict these outcomes through an intuitive process, and then monitor the outcomes through user-friendly charts.
For more information, see Predict customer needs and behaviors by using text predictions in your conversational channels.
Upgrade impact
Channels that you configured with text analyzers in the previous version of your system continue to work in the same manner after the upgrade to the current version. When you edit and save the configuration of an existing channel, the text analyzer rule is automatically upgraded to a text prediction. The associated text prediction is now an object where you can manage and monitor the text analytics for your channel. When you create a new channel in the upgraded system, the system automatically creates a text prediction for that channel.
What steps are required to update the application to be compatible with this change?
- Enable the asynchronous model building and reporting in text predictions through job schedulers that use the System Runtime Context (SRC) by adding your application to the SRC.
For more information, see Automating the runtime context management of background processes. - Enable model building in text predictions by configuring background processing nodes.
For more information, see Assigning decision management node types to Pega Platform nodes.
External data flow rules are removed
Valid from Pega Version 8.6
In previous versions of Pega Platform™, you could configure data flows to run in an external Hadoop environment. The external data flows functionality was deprecated and hidden from view in Pega Platform 8.5. The functionality has been now removed and is no longer available in Pega Platform 8.6.
For more information, see External data flow rules are deprecated.