Creating an HDFS data set record
You must configure each instance of the HDFS data set rule before it can read data from and save it to an external Apache Hadoop Distributed File System (HDFS).
- Create an instance of the HBase data set rule:
- In the header of Dev Studio, click .
- In the Label field, enter a short description for your data set.
- In the Type field, select HDFS.
- In the Context section, select the application context, class,
and ruleset for the data set.For more information about the fields on this form, see Creating rules.
- Click Create and Open.
- On the Create Data Set tab, in the Data Set Record
Configuration section, define the following settings to identify your data
set:
- In the Label field, enter the data set label.
- Optional: To change the automatically created identifier, click Edit, enter an identifier name, and then click OK.
- In the Type list, select HDFS.
- In the Context section, specify the application context, applicable class, ruleset, and ruleset version of the data set.
- Click Create and open.
- In the Connection section, connect to a Hadoop configuration
instance:
- In the Hadoop configuration instance field, select the
Hadoop configuration instance that contains the HDFS storage configuration.You can create a Hadoop configuration instance by clicking the Target icon on the right side of the field.
- Test whether Pega Platform can connect to the HDFS data set by
clicking Test connectivity.
- In the File path field, specify a file path to the group of
source and output files that the data set represents.
- In the Hadoop configuration instance field, select the
Hadoop configuration instance that contains the HDFS storage configuration.
- Optional: To view the first 100 KB of records in the selected file, click Preview file.
- In the Parser configuration section, specify the file type that
is used within the selected data set:
Options Actions CSV - In the File type field, select CSV.
- In the Delimiter character field, select the character for separating properties.
- In the Supported quotes field, select the type of quotation marks that you want to use.
JSON In the File type field, select JSON. Parquet - In the File type field, select Parquet.
- In the Compression algorithm field, for data set write
operations, specify the algorithm that is used for file compression in the data
set.
- If you do not use a file compression method in the data set, select Uncompressed.
- If you use the gzip file compression algorithm in your data set, select Gzip.
- If you use the SNAPPY file compression algorithm in your data set, select Snappy.
- In the Properties mapping section, map the properties from the
HDFS data set to the corresponding Pega Platform properties, depending
on your parser configuration:
Options Actions CSV - Click Add Property.
- In the numbered field, specify the property that corresponds to a column in
the CSV file.
JSON - To use the auto-mapping mode, keep the Use property auto mapping check box selected.
- To manually map properties, perform the following actions:
- Clear the Use property auto mapping check box.
- In the JSON column, enter the name of the column that you want to map to a Pega Platform property.
- In the Property name field, specify a Pega Platform property that you want to map to the JSON column.
Parquet - Click Generate missing properties.
- Examine the Properties generation dialog that shows both mapped and unmapped properties.
- Click Submit.
To create the mapping, Parquet utilizes properties that are defined in the data set class. You can map only the properties that are scalar and not inherited. If the property name matches a field name in the Parquet file, the property is populated with the corresponding data from the Parquet file.
You can generate properties from the Parquet file that do not exist in Pega Platform. When you generate missing properties, Pega Platform checks for unmapped columns in the data set, and creates the missing properties in the data set class for any unmapped columns.
- Click Save.
Previous topic File storage Next topic Configuring Hadoop settings for an HDFS connection