Changes to the architecture of the Data Flow service
Valid from Pega Version 8.4
In Pega Platform™ 8.4, the architecture of batch and real-time data flows uses improved node handling to increase the stability of data flow runs. As a result, there are fewer interactions with the database and between the nodes, resulting in increased resilience of the Data Flow service.
If you upgrade from a previous version of Pega Plaftorm, see the following list for an overview of the changes in the behavior of the Data Flow service compared to previous versions:
- Responsiveness
- Updates to lifecycle actions
- Starting a run
- Triggering pre- and post-activities
- Selecting a node fail policy
- No service nodes and active runs
Responsiveness
Nodes no longer communicate and trigger each other, but run periodic tasks instead. As such, triggering a new run does not cause the service nodes to immediately start the run. Instead, the run starts a few seconds later. The same applies to user actions such as stopping, starting, and updating the run. The system also processes topology changes as periodic tasks, so it might take a few minutes for new nodes to join runs, or for partitions to redistribute when a node leaves a run.
Updates to lifecycle actions
To make lifecycle actions more intuitive, the Stop action consolidates both the Stop and Pause actions. The Start action consolidates both the Resume and Start actions.
You can resume or restart stopped and failed runs with the Start and Restart actions. The Start action is only available for resumable runs and continues the run from where it stopped. The Restart action causes the run to process from the beginning. Completed runs can only be restarted. If a run completes with failures, you can restart it from the beginning, or process only the errors by using the Reprocess failures action.
Starting a run
New data flow runs have the Initializing status, and start automatically. You no longer need to manually start a new run, so the New status is now removed.
If there are no nodes available to process a run, the run gets the Queued status and waits for an available node.
Triggering pre- and post-activities
The system now triggers pre-activities on a random service node, rather than on the node that triggered the run.
The system triggers post-activities only for runs that complete, fail, or complete with failures. If you manually stop a run with the Stop action, the post-activity does not trigger. However, restarting the run with the Restart action triggers first the post-activity, and then the pre-activity.
You can no longer choose to run pre- and post-activities on all nodes.
Selecting a node fail policy
For resumable runs, you can no longer select a node fail policy. If a node fails, the partitions assigned to that node automatically continue the run on different nodes.
For non-resumable runs, you can choose to restart the partitions assigned to the failed node on different nodes, or to fail the partitions assigned to the failed node.
No service nodes and active runs
If the last data flow node for an in-progress run fails, the run remains in the In Progress state, even if no processing takes place. This behavior results from the fact that data flow architecture now prevents unrelated nodes from affecting runs.
Email Wizard support discontinued
Valid from Pega Version 8.4
Pega Platform™ no longer includes the Email Wizard. This wizard helped set up an email service for sending and receiving email in Pega Platform. The wizard generated an email account, an email listener, and an email service rule.
After an upgrade to Pega Platform 8.4 and later, existing clients must create new email accounts and email channels in App Studio. When you configure a new email channel, you add your email accounts so that customers can send and receive email by using the Pega Email Bot. Configuration of an email channel automatically generates an email listener and service email rules.
For more information, see Creating an email account and Building an Email channel.
Improved indexing performance by gradual retrieval of data
Valid from Pega Version 8.4
Search functionality in Pega Platform™ versions from 8.4.5 to 8.6 now includes the option to improve indexing performance when reading query results from a large table in a database. For example, to load the recommended 50 records at a time, in the Pega-SearchEngine ruleset, create the indexing/distributed/fetchsize dynamic system setting, and set the value to 50.
Creating the fetchsize dynamic system setting ensures that your system does not crash while indexing classes with numerous instances.
For more information, see Creating a dynamic system setting.