Configuring External Kafka as a Stream service
This content applies only to On-premises and Client-managed cloud environments
Configure the connection details between the Pega clusters and a Kafka service
running in your organization for a data streaming service. Use the appropriate dynamic
system settings in the prconfig.xml configuration file used by Pega Platform for these connection details.
Starting in Pega Platform version 8.7, the use of embedded Kafka (a Kafka service running on steam nodes in your deployment) is deprecated. Existing VM-based deployments of Pega Platform can continue to use Kafka in embedded mode, but will not future support software updates. To ensure future compatibility, do not create any new environments using embedded Kafka.
Enable external Kafka
To use external Kafka as stream service, configure the following prconfig settings in your environment:
<!-- Set stream provider as external Kafka -->
<env name="services/stream/provider" value="ExternalKafka"/>
<!-- Provide bootstrap servers -->
<env name="services/stream/broker/url" value="broker1:9092, broker2:9092"
/>
<!-- Optional: provide desired replication factor. By default, we will set
replication factor equal to the number of brokers in the previous setting
-->
<!-- Available from 8.4.1 onwards -->
<env name="services/stream/external/replication/factor" value="2"/>
Connection properties
To establish a secure connection between Pega nodes and your external Kafka cluster, the following connection properties can be set by using prconfigs:
<!-- Security protocol -->
<env name="services/stream/encryption/security/protocol" value="
PLAINTEXT|SSL|SASL_PLAINTEXT|SASL_SSL"/>
<!-- SSL certificate settings -->
<env name="services/stream/encryption/truststore/path" value="/path/to
/truststore.jsk"/>
<env name="services/stream/encryption/truststore/password" value="
my_passowrd"/>
<env name="services/stream/encryption/keystore/path" value="/path/to
/keystore.jks"/>
<env name="services/stream/encryption/keystore/password" value="
my_password"/>
<env name="services/stream/encryption/key/password" value="my_password"/>
<!-- SASL configuration. See supported mechanisms here: https://docs.
confluent.io/current/kafka/authentication_sasl/index.html -->
<env name="services/stream/encryption/sasl/mechanism" value="PLAIN"/>
<!-- It is important to escape any nested quotes with the " This
doesn't apply to Dynamic System Settings -->
<env name="services/stream/encryption/sasl/jaas/config" value="org.apache.
kafka.common.security.scram.ScramLoginModule required
username="my_user" password="my_password";"/>Topic name pattern
By default, topics originating from Pega Platform have the pega- prefix, so that it is easy to distinguish them from topics created by other applications. You can configure this pattern, to customize topic names per environment.
<env name="services/stream/name/pattern" value="pega-dev-{stream.name}"/>Example: Confluent cloud
In this example, you configure Pega stream service to point to https://confluent.cloud.
Follow the Quick Start for Apache Kafka using Confluent Cloud guide on the Confluent Documentation website to get your first Kafka cluster running. Once you have the cluster ready, copy the Kafka connection details. Here is an example of the connection details we are going to use:
bootstrap.servers=pkc-4nya7.us-east-1.aws.confluent.cloud:9092
security.protocol=SASL_SSL
sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule
required username="my_first_user" password="this_is_password";
sasl.mechanism=PLAINAdd the following prconfig settings on every Pega platform node.
For Pega Platform version 8.4.0 configurations, use:
<env name="services/stream/provider" value="ExternalKafka"/>
<!--
8.4.0 doesn't allow to configure replication factor. We use replication
factor equal to number of bootstrap servers as a default.
Therefore, you can copy the same bootstrap server many times to achieve
desired replication factor.
Starting from 8.4.1 we allow to configure desired replication factor
-->
<env name="services/stream/broker/url" value="pkc-4nya7.us-east-1.aws.
confluent.cloud:9092, pkc-4nya7.us-east-1.aws.confluent.cloud:9092, pkc4nya7.us-east-1.aws.confluent.cloud:9092"/>
<!-- security settings -->
<env name="services/stream/encryption/security/protocol" value="
PLAINTEXT|SSL|SASL_PLAINTEXT|SASL_SSL"/>
<env name="services/stream/encryption/sasl/mechanism" value="SASL_SSL"/>
<env name="services/stream/encryption/sasl/jaas/config" value="org.apache.
kafka.common.security.plain.PlainLoginModule required username="
my_first_user" password="this_is_password";"/>
For Pega Platform version 8.4.1+ configurations, use:
<env name="services/stream/provider" value="ExternalKafka"/>
<env name="services/stream/broker/url" value="pkc-4nya7.us-east-1.aws.
confluent.cloud:9092"/>
<!-- Confluent cloud requires replication factor to be set to 3 -->
<env name="services/stream/external/replication/factor" value="3"/>
<!-- security settings -->
<env name="services/stream/encryption/security/protocol" value="
PLAINTEXT|SSL|SASL_PLAINTEXT|SASL_SSL"/>
<env name="services/stream/encryption/sasl/mechanism" value="SASL_SSL"/>
<env name="services/stream/encryption/sasl/jaas/config" value="org.apache.
kafka.common.security.plain.PlainLoginModule required username="
my_first_user" password="this_is_password";"/>
All Pega nodes then need to be restarted.
Previous topic Understanding pr_data_stream_* tables Next topic Securing Kafka and Cassandra JMX