Creating a data flow
Create a data flow to process and move data between data sources. Customize your data flow by adding data flow shapes and by referencing other business rules to do more complex data operations. For example, a simple data flow can move data from a single data set, apply a filter, and save the results in a different data set. More complex data flows can be sourced by other data flows, can apply strategies for data processing, and open a case or trigger an activity as the final outcome of the data flow.
- In the header of Dev Studio, click .
- In the Create Data Flow tab, create the rule that stores the data
flow:
- In the header of Dev Studio, click .
- On the Create form, enter values in the fields to define the context of the flow.
- In the Label field, describe the purpose of the data flow.
- Optional: To change the default identifier for the data flow, click Edit, enter a meaningful name, and then click OK.
- In the Apply to field, press the Down arrow key, and then
select the class that defines the scope of the flow. The class controls which rules the data flow can use. It also controls which rules can call the data flow.
- In the Add to ruleset field, select the name and version of a ruleset that stores the data flow.
- Click Create and open.
- In the Edit Data flow tab, double-click the Source shape.
- In the Source configurations window, in the
Source list, define a primary data source for the data flow by
selecting one of the following options:
- To receive data from an activity or from a data flow with a destination that refers to your data flow, select Abstract.
- To receive data from a different data flow, select Data flow. Ensure that the data flow that you select has an abstract destination defined.
- To receive data from a data set, select Data set. If you
select a streaming data set, such as Kafka, Kinesis, or Stream, in the Read
options section, define a read option for the data flow:
- To read both real-time records and data records that are stored before the start of the data flow, select Read existing and new records.
- To read only real real-time records, select Only read new records.
For more information, see Data Set rule form - Completing Data Set tab.
- To retrieve and sort information from the PegaRULES database, an external database, or an Elasticsearch index, select Report definition.
- In the Source configurations window, click Submit.
- Optional: To facilitate data processing, transform data that comes from the data source by performing one or more of the following procedures:
- Optional: To apply advanced data processing on data that comes from the data source, call other rule types from the data flow by performing one or more of the following procedures:
- In the Edit Data flow tab, double-click the Destination shape.
- In the Destination configurations window, in the
Destination list, define the output point of the data flow by
selecting one of the following options:
- If you want other data flows to use your data flow as their source, select Abstract.
- If you want an activity to use the output data from your data flow, select Activity.
- If you want to start a case as the result of a completed data flow, select Case. The created case contains the output data from your data flow.
- If you want to send output data to a different data flow, select Data flow. Ensure that the data flow that you select has an abstract source defined.
- To save the output data into a data set, select Data
set.
For more information, see Data Set rule form - Completing Data Set tab.
- In the Source configurations window, click Submit.
- In the Edit data flow tab, click Save.
- Filtering incoming data
- Combining data from two sources
- Converting the class of the incoming data pages
- Merging data
- Applying complex data transformations
- Applying complex event processing
- Adding strategies to data flows
- Applying text analysis on the data records
- Branching a data flow
- Configuring a data flow to update a single property only
- Types of data flows
- Changing the number of retries for SAVE operations in batch and real-time data flow runs
- Adding pre- and post- activities to data flows
- Recording scorecard explanations through data flows
Previous topic Processing data with data flows Next topic Filtering incoming data