Cluster ID
Kafka has an internal check to ensure that a broker cannot join a different cluster to the one in which it was previously registered (this is important to avoid various kinds of metadata inconsistencies in the cluster). The clusterId is stored locally after initial registration and is verified upon cluster startup that it still matches what is in ZooKeeper.
This clusterId is stored in the meta.properties file in the folder specified by the log.dirs setting: this is persisted on a PVC created by Kafka. This PVC is not removed when the Kafka ZNode is deleted, which means that there are circumstances where this internal check fails with the following error:
The Cluster ID <new Cluster ID> doesn't match stored clusterId <old Cluster ID> in meta.properties.Possible scenarios
Restarting a Kafka cluster
When re-starting a Kafka cluster, ensure that the Kafka ZNode is not removed: upon restart the cluster attempts to register with the ZooKeeper cluster referenced in the ZNode and checks that the cluster IDs match. As the meta.properties file has not been changed this should not cause any problems.
Replacing an existing ZNode
If the ZNode has been removed, then the Kafka PVC prefixed with log-dirs- also has to be removed.
This results in the loss of topic metadata but is unavoidable since Kafka needs to re-register with ZooKeeper anyway.
For instance, this applies when breaking changes have been made to the ZooKeeper operator.