You can now transfer data between cloud-based storage systems and your applications in the cloud by using data flows. Instead of creating and maintaining relational databases to transfer data to or from remote files, you can add a reference to a remote repository in a File data set. The files may contain information, such as customer data, purchase lists, or offer details, that you can apply to your applications by using parallel loads from CSV or JSON files, including compressed ZIP and GZIP archives.

The following figure presents the configuration of remote repositories:

Data set configuration - remote repositories

Configuring a File data set with a remote repository connection allows you to automatically process that data in various scenarios. For example, by referencing that File data set from a data flow in a decisioning strategy for promotional offers, you can automatically download a detailed list of customers that is stored in a collection of GZIP archives in a Microsoft Azure cloud directory and let the strategy model select the clients who receive new discount offers.

By using the updated data set interface, you can map Pega Platform properties to the relevant columns in the remote file:

Data set configuration - mapping properties

See the following video for a sample data set creation process:

Apart from sourcing data, you can also configure an automatic export of data to a set of remote files at the same time through wildcards.

Currently, Pega Platform supports connections with the following repositories:

• JFrog Artifactory
• Amazon S3
• Microsoft Azure
• local file system

For more information, see:

• Tutorial: Configuring a remote repository as a source for a File data set
• Tutorial: Referencing remote repository data in a data flow
• Creating a File data set record for files on repositories