Configuring External Kafka as a Stream service
This content applies only to On-premises and Client-managed cloud environments
Configure external Kafka as a stream service provider to use your own managed Kafka infrastructure.
You can configure external Kafka either by setting relevant properties on every Pega Platform node, or by using dynamic system settings, which are applied globally. For more information, see How to compose the key of a PRCONFIG dynamic system setting.
Enable external Kafka
To use external Kafka as stream service, configure the following prconfig settings in your environment:
<!-- Set stream provider as external Kafka -->
<env name="services/stream/provider" value="ExternalKafka"/>
<!-- Provide bootstrap servers -->
<env name="services/stream/broker/url" value="broker1:9092, broker2:9092"
/>
<!-- Optional: provide desired replication factor. By default, we will set
replication factor equal to the number of brokers in the previous setting
-->
<!-- Available from 8.4.1 onwards -->
<env name="services/stream/external/replication/factor" value="2"/>
Connection properties
To establish a secure connection between Pega nodes and your external Kafka cluster, the following connection properties can be set by using prconfigs:
<!-- Security protocol -->
<env name="services/stream/encryption/security/protocol" value="
PLAINTEXT|SSL|SASL_PLAINTEXT|SASL_SSL"/>
<!-- SSL certificate settings -->
<env name="services/stream/encryption/truststore/path" value="/path/to
/truststore.jsk"/>
<env name="services/stream/encryption/truststore/password" value="
my_passowrd"/>
<env name="services/stream/encryption/keystore/path" value="/path/to
/keystore.jks"/>
<env name="services/stream/encryption/keystore/password" value="
my_password"/>
<env name="services/stream/encryption/key/password" value="my_password"/>
<!-- SASL configuration. See supported mechanisms here: https://docs.
confluent.io/current/kafka/authentication_sasl/index.html -->
<env name="services/stream/encryption/sasl/mechanism" value="PLAIN"/>
<!-- It is important to escape any nested quotes with the " This
doesn't apply to Dynamic System Settings -->
<env name="services/stream/encryption/sasl/jaas/config" value="org.apache.
kafka.common.security.scram.ScramLoginModule required
username="my_user" password="my_password";"/>Topic name pattern
By default, topics originating from Pega Platform have the pega- prefix, so that it is easy to distinguish them from topics created by other applications. You can configure this pattern, to customize topic names per environment.
<env name="services/stream/name/pattern" value="pega-dev-{stream.name}"/>Example: Confluent cloud
In this example, you configure Pega stream service to point to https://confluent.cloud.
Follow the Quick Start for Apache Kafka using Confluent Cloud guide on the Confluent Documentation website to get your first Kafka cluster running. Once you have the cluster ready, copy the Kafka connection details. Here is an example of the connection details we are going to use:
bootstrap.servers=pkc-4nya7.us-east-1.aws.confluent.cloud:9092
security.protocol=SASL_SSL
sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule
required username="my_first_user" password="this_is_password";
sasl.mechanism=PLAINAdd the following prconfig settings on every Pega platform node.
For Pega Platform version 8.4.0 configurations, use:
<env name="services/stream/provider" value="ExternalKafka"/>
<!--
8.4.0 doesn't allow to configure replication factor. We use replication
factor equal to number of bootstrap servers as a default.
Therefore, you can copy the same bootstrap server many times to achieve
desired replication factor.
Starting from 8.4.1 we allow to configure desired replication factor
-->
<env name="services/stream/broker/url" value="pkc-4nya7.us-east-1.aws.
confluent.cloud:9092, pkc-4nya7.us-east-1.aws.confluent.cloud:9092, pkc4nya7.us-east-1.aws.confluent.cloud:9092"/>
<!-- security settings -->
<env name="services/stream/encryption/security/protocol" value="
PLAINTEXT|SSL|SASL_PLAINTEXT|SASL_SSL"/>
<env name="services/stream/encryption/sasl/mechanism" value="SASL_SSL"/>
<env name="services/stream/encryption/sasl/jaas/config" value="org.apache.
kafka.common.security.plain.PlainLoginModule required username="
my_first_user" password="this_is_password";"/>
For Pega Platform version 8.4.1+ configurations, use:
<env name="services/stream/provider" value="ExternalKafka"/>
<env name="services/stream/broker/url" value="pkc-4nya7.us-east-1.aws.
confluent.cloud:9092"/>
<!-- Confluent cloud requires replication factor to be set to 3 -->
<env name="services/stream/external/replication/factor" value="3"/>
<!-- security settings -->
<env name="services/stream/encryption/security/protocol" value="
PLAINTEXT|SSL|SASL_PLAINTEXT|SASL_SSL"/>
<env name="services/stream/encryption/sasl/mechanism" value="SASL_SSL"/>
<env name="services/stream/encryption/sasl/jaas/config" value="org.apache.
kafka.common.security.plain.PlainLoginModule required username="
my_first_user" password="this_is_password";"/>
All Pega nodes then need to be restarted.
Previous topic Understanding pr_data_stream_* tables Next topic Securing Kafka and Cassandra JMX