Creating a data flow

Create a data flow to process and move data between data sources. Customize your data flow by adding data flow shapes and referencing other business rules to do more complex data operations. For example, a simple data flow can move data from a single data set, apply a filter, and save the results in other data set. More complex data flows can be sourced by other data flows, apply strategies for data processing, and open a case or trigger an activity as the final outcome of the data flow.

1. Create the rule that stores the data flow.
   1. In the header of Dev Studio, click Create > Data Model > Data Flow.
   2. On the Create form, enter values in the fields to define the context of the flow.
   3. In the Label field, enter text that describes the purpose of the data flow.
   4. Optional: To change the default identifier for the data flow, click Edit, and then provide a unique value in the Identifier field.
   5. In the Apply to field, press the Down Arrow key and select the class that defines the scope of the flow.
      The class controls which rules the data flow can use. It also controls which rules can call the data flow.
   6. In the Add to ruleset field, select the name and version of a ruleset that stores the data flow.
   7. Click Create and open to open the Data Flow form.
2. In the Data Flow tab, double-click the Source shape, and select the data source for your data flow. Each data flow contains one primary source that is a standard entry point of a data flow. The primary source can originate from.
   • Abstract - Set the source to Abstract when you want to receive data from an activity or from a data flow with destination that refers to your data flow.
   • Data flow - Set the source to Data flow to receive data from other data flow. To use a data flow as the primary input the data flow to which you refer must have abstract destination.
   • Data set - Set the source to Data set to receive data from a data set.
     For streaming data sets (Kafka, Kinesis, Stream), select one of the read options:

     • Read existing and new records - When the data flow has started, it receives past data records that the streaming data set stores and the real-time records from the streaming data. This is a default option.
     • Only read new records - When the data flow has started, the data flow receives real-time data records from the streaming data.

     For more information, see Data Set rule form - Completing Data Set tab.

   • Report definition - Set the source to Report definition to retrieve and sort information from the PegaRULES database, an external database, or an Elasticsearch index.
   Note: Secondary sources appear in the Data Flow tab, when you start combining and merging data. Secondary sources can originate from a data set, data flow, or report definition.
3. Optional: To facilitate data processing, transform data that comes from the data source.
   • Filter incoming data.
   • Combine data from two sources into a page or page list.
   • Convert the class of the incoming data pages.
   • Merge data.
4. Optional: To apply advanced data processing on data that comes from the data source, call other rule types from the data flow.
   • Apply complex data transformations.
   • Apply event strategies for complex event processing.
   • Apply strategies for improving results of customer interactions.
   • Apply text analysis on the records that contain text.
5. Double-click the Destination shape and select the output point of your data flow.
   • Abstract - Set the destination to Abstract when you want other data flow to use your data flow as its source.
   • Activity - Set the destination to Activity when you want to configure an activity to use the output data from your data flow.
   • Case - Set the destination to Case when you want to start a case as the result of your data flow. The case can contain the output data from your data flow.
   • Data flow - Set the destination to Data flow to send data to another data flow. This data flow must have abstract source to use the ouput data of your data flow.
   • Data set - Set the destination to Data set to save data into a data set. For more information, see Data Set rule form - Completing Data Set tab.
     Note: You cannot save data into Monte Carlo, Stream, or social media data sets.