Stackable Operator for Apache Kafka

Follow the Getting started which will guide you through installing The Stackable Kafka and ZooKeeper operators, setting up ZooKeeper and Kafka and testing your Kafka using kcat.

The KafkaCluster custom resource contains your Kafka cluster configuration. It defines a single broker role.

For every role group in the broker role the operator creates a StatefulSet. Multiple Services are created - one at role level, one per role group as well as one for every individual Pod - to allow access to the entire Kafka cluster, parts of it or just individual brokers.

For every StatefulSet (role group) a ConfigMap is deployed containing a log4j.properties file for logging configuration and a server.properties file containing the whole Kafka configuration which is derived from the KafkaCluster resource.

The operator creates a Service discovery ConfigMap for the whole KafkaCluster which references the Service for the whole cluster. Other operators use this ConfigMap to connect to a Kafka cluster simply by name and it can also be used by custom third party applications to find the connection endpoint.

Kafka requires Apache ZooKeeper for coordination purposes (it will not be needed in the future as it will be replaced with a built-in solution).

Since Kafka often takes on a bridging role, many other products connect to it. In the demos below you will find example data pipelines that use Apache NiFi with the Stackable operator to write to Kafka and Apache Druid with the Stackable operator to read from Kafka. But you can also connect using Apache Spark or with a custom Job written in various languages.

stackablectl supports installing Demos with a single command. The demos are complete data piplines which showcase multiple components of the Stackable platform working together and which you can try out interactively. Both demos below inject data into Kafka using NiFi and read from the Kafka topics using Druid.

The Stackable operator for Apache Kafka currently supports the Kafka versions listed below. To use a specific Kafka version in your KafkaCluster, you have to specify an image - this is explained in the Product image selection documentation. The operator also supports running images from a custom registry or running entirely customized images; both of these cases are explained under Product image selection as well.