Configuring Hadoop settings for an HDFS connection
Use the HDFS settings in the Hadoop data instance to configure connection details for the HDFS data sets.
By using the Hadoop infrastructure, you can process large amounts of data directly on the Hadoop cluster and reduce the data transfer between the Hadoop cluster and the Pega Platform. Hadoop configuration instances are records in the SysAdmin category and belong to the Data-Admin-Hadoop class.
- In the header of Dev Studio, click .
- On the Create Hadoop form, enter a description and a name for the Hadoop data instance.
- Click Create and open.
- In the Connection section, specify a master Hadoop host.This host must contain HDFS NameNode and HBase master node.
- In the HDFS section, select the Use HDFS configuration check box.
- In the User name field, enter the user name to authenticate in HDFS.
- In the Port field, enter the port of the HDFS NameNode.The default port is 8020.
- Optional: To specify a custom HFDS NameNode, select the Advanced
configuration check box.
- In the Namenode field, specify a custom HDFS NameNode that is different from the one defined in the common configuration.
- In the Response timeout field, enter the number of
milliseconds to wait for the server response. Enter zero or leave it empty to wait
The default timeout is 3000.
- In the KMS URI field, specify an instance of Hadoop Key
Management Server to access encrypted files from the Hadoop server.
For example, for a KMS server running on
http://localhost:16000/kms, the KMS URI is
- Optional: To enable secure connections, select the Use authentication
Note: To authenticate with Kerberos, you must configure your environment. For more details, see the Kerberos Network Authentication Protocol documentation.
- In the Master kerberos principal field, enter the Kerberos principal name of the HDFS NameNode as defined and authenticated in the Kerberos Key Distribution Center, typically following the nn/<hostname>@<REALM> pattern.
- In the Client kerberos principal field, enter the Kerberos principal name of a user as defined in Kerberos, typically in the following format: <username>/<hostname>@<REALM>.
- In the Keystore field, enter the name of a keystore that
contains a keytab file with the keys for the user who is defined in the Client Kerberos
Note: The keytab file is in a readable location on the Pega Platform server, for example: /etc/hdfs/conf/thisUser.keytab or c:\authentication\hdfs\conf\thisUser.keytab.
- Test the connection to the HDFS NameNode by clicking Test connectivity.
- Click Save.
Previous topic Creating an HDFS data set record Next topic Creating a File data set record for embedded files