Creating an HDFS data set record

You must configure each instance of the HDFS data set rule before it can read data from and save it to an external Apache Hadoop Distributed File System (HDFS).

  1. Create an instance of the HDFS data set rule.
  2. Connect to an instance of the Data-Admin-Hadoop configuration rule.
    1. In the Hadoop configuration instance field, reference the Data-Admin-Hadoop configuration rule that contains the HDFS storage configuration.
    2. Click Test connectivity to test whether Pega Platform can connect to the HDFS data set.
      Note: The HDFS data set is optimized to support connections to one Apache Hadoop environment. When HDFS data sets connect to different Apache Hadoop environments in the single instance of a data flow rule, the data sets cannot use authenticated connections concurrently. If you need to use authenticated and non-authenticated connections at the same time, the HDFS data sets must use one Hadoop environment.
  3. In the File path field, specify a file path to the group of source and output files that the data set represents.
    Note: This group of files is based on the file within the original path, but also contains all of the files with the following pattern: fileName-XXXXX, where XXXXX are sequence numbers starting from 00000. This is a result of data flows saving records in batches. The save operation appends data to the existing HDFS data set without overwriting it. You can use * to match multiple files in a folder (for example, /folder/part-r-* ).
  4. Optional: Click Preview file to view the first 100 KB of records in the selected file.
  5. In the File format section, select the file type that is used within the selected data set.

    • CSV

      If your HDFS data set uses the CSV file format, you must specify the following properties for content parsing within the Pega Platform :

      • The delimiter character for separating properties
      • The supported quotation marks
    • JSON

    • Parquet

      For data set write operations, specify the algorithm that is used for file compression in the data set:

      • Uncompressed - Select this option if you do not use a file compression method in the data set.
      • Gzip - Select this option if you use the GZIP file compression algorithm in your data set.
      • Snappy - Select this option if you use the SNAPPY file compression algorithm in your data set.
  6. In the Properties mapping section, map the properties from the HDFS data set to the corresponding Pega Platform properties, depending on your parser configuration.

    • CSV

      1. Click Add Property.
      2. In the numbered field that is displayed, specify the property that corresponds to a column in the CSV file.
      Note: Property mapping for the CSV format is based on the order of columns in the CSV file. For that reason, the order of the properties in the Properties mapping section must correspond to the order of columns in the CSV file.

    • JSON

      • To use the auto-mapping mode, select the Use property auto mapping check box. This mode is enabled by default.
      • To manually map properties:
        1. Clear the Use property auto mapping check box.
        2. In the JSON column, enter the name of the column that you want to map to a Pega Platform property.
        3. In the Property name field, specify a Pega Platform property that you want to map to the JSON column.

        In auto-mapping mode, the column names from the JSON data file are used as Pega Platform properties. This mode supports the nested JSON structures that are directly mapped to Page and Page List properties in the data model of the class that the data set applies to.

    • Parquet

      To create the mapping, Parquet utilizes properties that are defined in the data set class. You can map only the properties that are scalar and not inherited. If the property name matches a field name in the Parquet file, the property is populated with the corresponding data from the Parquet file.

      You can generate properties from the Parquet file that do not exist in Pega Platform. When you generate missing properties, Pega Platform checks for unmapped columns in the data set, and creates the missing properties in the data set class for any unmapped columns.

      To generate missing properties:
      1. Click Generate missing properties.
      2. Examine the Properties generation dialog that shows both mapped and unmapped properties.
      3. Click Submit to generate the unmapped properties.
  7. Click Save.