KRaft mode (experimental)
The Kafka KRaft mode is currently experimental, and subject to change. |
Apache Kafka’s KRaft mode replaces Apache ZooKeeper with Kafka’s own built-in consensus mechanism based on the Raft protocol. This simplifies Kafka’s architecture, reducing operational complexity by consolidating cluster metadata management into Kafka itself.
The Stackable Operator for Apache Kafka currently does not support automatic cluster upgrades from Apache ZooKeeper to KRaft. |
Overview
-
Introduced: Kafka 2.8.0 (early preview, not production-ready).
-
Matured: Kafka 3.3.x (production-ready, though ZooKeeper is still supported).
-
Default & Recommended: Kafka 3.5+ strongly recommends KRaft for new clusters.
-
Full Replacement: Kafka 4.0.0 (2025) removes ZooKeeper completely.
-
Migration: Tools exist to migrate from ZooKeeper to KRaft, but new deployments should start with KRaft.
Configuration
The Stackable Kafka operator introduces a new role in the KafkaCluster CRD called KRaft Controller
.
Configuring the Controller
will put Kafka into KRaft mode. Apache ZooKeeper will not be required anymore.
apiVersion: kafka.stackable.tech/v1alpha1
kind: KafkaCluster
metadata:
name: kafka
spec:
image:
productVersion: "3.9.1"
brokers:
roleGroups:
default:
replicas: 1
controllers:
roleGroups:
default:
replicas: 3
Using spec.controllers is mutually exclusive with spec.clusterConfig.zookeeperConfigMapName .
|
Recommendations
A minimal KRaft setup consisting of at least 3 Controllers has the following resource requirements:
-
600m
CPU request -
3000m
CPU limit -
3000Mi
memory request and limit -
6Gi
persistent storage
The Controller replicas should sum up to an odd number for the Raft consensus. |
Resources
Corresponding to the values above, the operator uses the following resource defaults:
controllers:
config:
resources:
memory:
limit: 1Gi
cpu:
min: 250m
max: 1000m
storage:
logDirs:
capacity: 2Gi
Overrides
The configuration of overrides, JVM arguments etc. is similar to the Broker and documented on the concepts page.
Internal operator details
KRaft mode requires major configuration changes compared to ZooKeeper:
-
cluster-id
: This is set to themetadata.name
of the KafkaCluster resource during initial formatting -
node.id
: This is a calculated integer, hashed from therole
androlegroup
and addedreplica
id. -
process.roles
: Will always only bebroker
orcontroller
. Mixedbroker,controller
servers are not supported. -
The operator configures a static voter list containing the controller pods. Controllers are not dynamicaly managed.
Known Issues
-
Automatic migration from Apache ZooKeeper to KRaft is not supported.
-
Scaling controller replicas might lead to unstable clusters.
-
Kerberos is currently not supported for KRaft in all versions.
Troubleshooting
Frequent leader elections
Likely caused by controller resource starvation or unstable Kubernetes scheduling.
Migration issues (ZooKeeper to KRaft)
Ensure Kafka version 3.9.x and higher and follow the official migration documentation. The Stackable Kafka operator currently does not support the migration.
Scaling issues
The Dynamic scaling is only supported from Kafka version 3.9.0. If you are using older versions, automatic scaling may not work properly (e.g. adding or removing controller replicas).