Creating a Kafka configuration instance
To manage connections to your Apache Kafka server or cluster of servers that is the source of your application stream data, configure a Kafka configuration instance in the Pega Platform Data-Admin-Kafka class.
With this configuration instance, you can then create one or more Kafka data sets in your application to stream data in real time. This instance ensures that you have stable, reliable connections to your Kafka server or cluster of servers and the specific topics that are part of your Kafka cluster.
Pega Platform supports Apache Kafka cluster version 0.10.0.1 or later.
To ensure that your application continuously processes your stream data, you can dedicate some background processing solely to Kafka stream data processing. If your system is deployed in Pega Cloud, the Pega Cloud services team configures this for you. If your system is deployed on premises, you can configure nodes to process Kafka stream data. For more information, see Deploying and operating the Stream service.
Depending on the type of your deployment, the following authentication methods are supported:
|Authentication method||Pega Cloud services deployments||Deployments on premises|
|SASL using a username and password||✓||✓|
|SASL using Kerberos||Not available||✓|
In systems deployed on premises, Pega supports configuring SASL authentication between Pega Platform and the Kafka cluster using a JAAS configuration file. To configure SASL authentication, perform the following steps:
- In the Kafka cluster, configure the Kafka Client credentials in the JAAS configuration file to enable either simple authentication (using a username and password) or Kerberos authentication.
- Pass the location of the JAAS configuration file as a JVM parameter in the Kafka cluster, for example: -Djava.security.auth.login.config = <path_to_JAAS_file>
For more information about configuring the JAAS configuration file, see the Apache Kafka documentation.
To create your Kafka configuration instance, perform the following steps:
- In the header of Dev Studio, click .
- On the New tab, enter identifying information for this
- In the Short description field, enter a comment on the purpose of this rule.
- In the Kafka field, enter an appropriate name for this Kafka service connection, for example, Kafka-service-1.
- Click Create and open.
- In the Details section, configure a host and port combination to
connect to the Kafka cluster:
- In the Host field, enter the address of the Kafka cluster.
- In the Port field, enter the port number.
- Optional: Click Add host to configure additional host and port combinations.
Note: Pega Platform discovers all nodes in the cluster during the first connection. This means that you can enter a single host and port combination to connect to a Kafka cluster. As a best practice, enter at least two host and port combinations to ensure a successful connection when a node is unavailable during a Pega Platform restart.
- Configure an authentication method for this Kafka server connection:
Options Actions SSL-based authentication
- In the Security settings section, select the Use SSL configuration check box.
- In the Truststore field, press the Down Arrow key and select a truststore file that contains a Kafka certificate or create a truststore record by clicking the Open icon.
- Select Use client certificate and enter the Pega Platform private key and private key password credentials in the Keystore and Key password fields respectively.
- In the Authentication section, select Use authentication.
- Select the authentication type:
- To enable authentication using login credentials, select Username and password, and then enter the login credentials.
- To enable authentication using Kerberos, select Kerberos, and then enter the Kerberos authentication key.
Note: Authentication using Kerberos is only supported in on-premises systems. Tip: If you see the message No JAAS configuration file set, SASL authentication between Pega Platform and the Kafka cluster is not configured. For configuration steps, see the Before you begin section of this procedure.
- Optional: To upload a client properties file containing the properties that you want to use to
establish the connection with the Kafka cluster, in the Advanced
configuration section, click Upload client
Note: The client properties file can contain the following properties:
- Click Test connectivity to test the connection between Pega Platform and the Kafka cluster.
- If the Kafka cluster is connected, click Save.
Previous topic Kafka data instances Next topic Integration-Services category