Skip to main content


         This documentation site is for previous versions. Visit our new documentation site for current releases.      
 

This content has been archived and is no longer being updated.

Links may not function; however, this content may be relevant to outdated versions of the product.

Configuring External Kafka as a Stream service

Updated on March 11, 2021

This content applies only to On-premises and Client-managed cloud environments

Configure external Kafka as a stream service provider to use your own managed Kafka infrastructure.

You can configure external Kafka either by setting relevant properties on every Pega Platform node, or by using dynamic system settings, which are applied globally. For more information, see How to compose the key of a PRCONFIG dynamic system setting.

Enable external Kafka

To use external Kafka as stream service, configure the following prconfig settings in your environment:

<!-- Set stream provider as external Kafka -->
<env name="services/stream/provider" value="ExternalKafka"/>
<!-- Provide bootstrap servers -->
<env name="services/stream/provider/url" value="broker1:9092, broker2:9092"
/>
<!-- Optional: provide desired replication factor. By default, we will set
replication factor equal to the number of brokers in the previous setting
-->
<!-- Available from 8.4.1 onwards -->
<env name="services/stream/external/replication/factor" value="2"/>

Connection properties

To establish a secure connection between Pega Platform nodes and your external Kafka cluster, the following connection properties can be set by using prconfigs:

<!-- Security protocol -->
<env name="services/stream/encryption/security/protocol" value="
PLAINTEXT|SSL|SASL_PLAINTEXT|SASL_SSL"/>
<!-- SSL certificate settings -->
<env name="services/stream/encryption/truststore/path" value="/path/to
/truststore.jsk"/>
<env name="services/stream/encryption/truststore/password" value="
my_passowrd"/>
<env name="services/stream/encryption/keystore/path" value="/path/to
/keystore.jks"/>
<env name="services/stream/encryption/keystore/password" value="
my_password"/>
<env name="services/stream/encryption/key/password" value="my_password"/>
<!-- SASL configuration. See supported mechanisms here: https://docs.
confluent.io/current/kafka/authentication_sasl/index.html -->
<env name="services/stream/encryption/sasl/mechanism" value="PLAIN"/>
<!-- It is important to escape any nested quotes with the " This
doesn't apply to Dynamic System Settings -->
<env name="services/stream/encryption/sasl/jaas/config" value="org.apache.
kafka.common.security.scram.ScramLoginModule required
username="my_user" password="my_password";"/>

Topic name pattern

By default, topics originating from Pega Platform have the pega- prefix, so that it is easy to distinguish them from topics created by other applications. You can configure this pattern, to customize topic names per environment.

<env name="services/stream/name/pattern" value="pega-dev-{stream.name}"/>
Important: Make sure that every Pega Platform node is configured with the same pattern. You can set the value for a pattern by using Dynamic System Settings. For more information, see How to compose the key of a PRCONFIG dynamic system setting.

Example: Confluent cloud

In this example, you configure Pega Platform stream service to point to https://confluent.cloud.

Follow the Quick Start for Apache Kafka using Confluent Cloud guide on the Confluent Documentation website to get your first Kafka cluster running. Once you have the cluster ready, copy the Kafka connection details. Here is an example of the connection details we are going to use:

bootstrap.servers=pkc-4nya7.us-east-1.aws.confluent.cloud:9092
security.protocol=SASL_SSL
sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule
required username="my_first_user" password="this_is_password";
sasl.mechanism=PLAIN

Add the following prconfig settings on every Pega Platform node.

For Pega Platform version 8.4.0 configurations, use:

<env name="services/stream/provider" value="ExternalKafka"/>
<!--
8.4.0 doesn't allow to configure replication factor. We use replication
factor equal to number of bootstrap servers as a default.
Therefore, you can copy the same bootstrap server many times to achieve
desired replication factor.
Starting from 8.4.1 we allow to configure desired replication factor
-->
<env name="services/stream/provider/url" value="pkc-4nya7.us-east-1.aws.
confluent.cloud:9092, pkc-4nya7.us-east-1.aws.confluent.cloud:9092, pkc4nya7.us-east-1.aws.confluent.cloud:9092"/>
<!-- security settings -->
<env name="services/stream/encryption/security/protocol" value="
PLAINTEXT|SSL|SASL_PLAINTEXT|SASL_SSL"/>
<env name="services/stream/encryption/sasl/mechanism" value="SASL_SSL"/>
<env name="services/stream/encryption/sasl/jaas/config" value="org.apache.
kafka.common.security.plain.PlainLoginModule required username="
my_first_user" password="this_is_password";"/>

For Pega Platform version 8.4.1+ configurations, use:

<env name="services/stream/provider" value="ExternalKafka"/>
<env name="services/stream/provider/url" value="pkc-4nya7.us-east-1.aws.
confluent.cloud:9092"/>
<!-- Confluent cloud requires replication factor to be set to 3 -->
<env name="services/stream/external/replication/factor" value="3"/>
<!-- security settings -->
<env name="services/stream/encryption/security/protocol" value="
PLAINTEXT|SSL|SASL_PLAINTEXT|SASL_SSL"/>
<env name="services/stream/encryption/sasl/mechanism" value="SASL_SSL"/>
<env name="services/stream/encryption/sasl/jaas/config" value="org.apache.
kafka.common.security.plain.PlainLoginModule required username="
my_first_user" password="this_is_password";"/>

All Pega Platform nodes then need to be restarted.

Have a question? Get answers now.

Visit the Support Center to ask questions, engage in discussions, share ideas, and help others.

Did you find this content helpful?

Want to help us improve this content?

We'd prefer it if you saw us at our best.

Pega.com is not optimized for Internet Explorer. For the optimal experience, please use:

Close Deprecation Notice
Contact us