Graceful shutdown

You can configure the graceful shutdown as described in Graceful shutdown.

Brokers

As a default, Kafka brokers have 30 minutes to shut down gracefully.

The Kafka broker process receives a SIGTERM signal when Kubernetes wants to terminate the Pod. After the graceful shutdown timeout runs out, and the process is still running, Kubernetes issues a SIGKILL signal.

This is equivalent to executing the bin/kafka-server-stop.sh command, which internally executes kill <kafka-pid> (code).

The broker logs the received signal as shown in the log below:

[2023-11-06 09:16:12,340] INFO Terminating process due to signal SIGTERM (org.apache.kafka.common.utils.LoggingSignalHandler)
[2023-11-06 09:16:12,341] INFO [KafkaServer id=1001] shutting down (kafka.server.KafkaServer)
[2023-11-06 09:16:12,346] INFO [KafkaServer id=1001] Starting controlled shutdown (kafka.server.KafkaServer)
[2023-11-06 09:16:12,379] INFO [Controller id=1001] Shutting down broker 1001 (kafka.controller.KafkaController)

Implementation

The Kafka documentation does a very good job at explaining what happens during a graceful shutdown:

The Kafka cluster automatically detects any broker shutdown or failure and elect new leaders for the partitions on that machine. This occurs if either a server fails or it is brought down intentionally for maintenance or configuration changes. For the latter cases Kafka supports a more graceful mechanism for stopping a server than just killing it. When a server is stopped gracefully it has two optimizations it takes advantage of:

  1. It syncs all its logs to disk to avoid the need for any log recovery when it restarts (i.e. validating the checksum for all messages in the tail of the log). Log recovery takes time, so this speeds up intentional restarts.

  2. It migrates any partitions the broker is the leader of, to other replicas prior to shutting down. This makes the leadership transfer faster and minimize the time each partition is unavailable to a few milliseconds.

Note that controlled shutdown only succeeds if all the partitions hosted on the broker have replicas (i.e. the replication factor is greater than 1 and at least one of these replicas is alive). This is generally what you want since shutting down the last replica would make that topic partition unavailable.

This operator takes care of that by only allowing a certain number of brokers to be offline as described in Allowed Pod disruptions. It also always set controlled.shutdown.enable=true explicitly.