Kafka with Charlatan
Kafka depends on the Zookeeper client library and the Zookeeper driver library; it uses the API of both libraries. The Zookeeper client library depends on the Zookeeper driver.
You can substitute the Zookeper library with the Charlatan library - a server that accepts and handles Zookeeper requests using the Zookeeper messaging protocol.
Charlatan consists of the following sub-projects:
- Charlatan-common: contains watch management, session management, common data access object (DAO) interfaces
- Charlatan-server: contains Zookeeper protocol implementation, Netty-based implementation of the server that accepts and handles zookeeper requests
Charlatan provides the following functionalities:
- Stores the node tree. See Node tree support.
- Stores and handles broker watches. See Broker watches support.
- Handles ephemeral nodes. See Ephemeral nodes support.
Node tree support
In order to support create, read, update, and delete (CRUD) operations on nodes, Charlatan DAO interface implementations use the Pega Platform database. Charlatan stores the node tree in the following table:
| nodeId | node ID |
| parentId | node parent ID |
| data | node data |
| mode | persistent/ephemeral |
| session | session which created the node |
| ... | fields with other node properties |
Charlatan node tree
Broker watches support
In order to support watches, all updates need to be stored in the database. The existing interface functionality can also be implemented by using Apache Ignite/Hazelcast, or other databases.
Once a broker sets watch on a specific node change, it should receive a one-time update, regardless of whether the change was performed by this or any other broker. Every broker pulls updates created by different brokers from the table and notifies its watches.
For every node CRUD operation, an entry in the following node updates table is created and stored:
| id | update ID |
| type | event type, for example: Created, Deleted, ChildrenChanged, DataChanged |
| path | full node path |
| broker | broker identifier that generated the change |
| timestamp | event time |
The node updates table
Ephemeral nodes support
If a node was created as ephemeral, it should be removed once the broker is off. An ephemeral broker from a previous broker session prevents the alive broker from starting. Removal of ephemeral nodes triggers alive brokers to re-select the leader, if needed, and to start replication operations.
The database is used to monitor broker session states. With every broker’s keepalive message, the last_seen timestamp of the session is updated. See the following pr_data_streamservice_sessions table:
| id | broker ID |
| session | broker session identifier |
| last_seen | timestamp of the last ping |
The broker sessions table
Stale sessions
Every Charlatan server periodically monitors its stale sessions and other Charlatan servers' stale sessions. If a stale session (last_seen timestamp timed out) is found, the server invalidates the stale session; it removes ephemeral nodes created by that session and notifies all connected brokers that are interested in this event.
When a Charlatan server/Pega Platform node crashes, broker sessions are invalidated by running the Charlatan server. If there are no more running Charlatan servers, the sessions are invalidated during the Charlatan server startup.
When a Kafka broker crashes, the Charlatan server invalidates its session.
To view the main outline for this article, see Kafka as a streaming service.
Previous topic Kafka standard deployment Next topic Monitoring Kafka