Graceful shutdown

You can configure the graceful shutdown as described in Graceful shutdown.

Servers

As a default, OPA servers have 2 minutes to shut down gracefully.

The OPA server process will receive a SIGTERM signal when Kubernetes wants to terminate the Pod. It will acknowledge the shutdown as shown in the log below and initiate a graceful shutdown. After the graceful shutdown timeout runs out, and the process still didn’t exit, Kubernetes will issue a SIGKILL signal.

{"level":"info","msg":"Shutting down...","time":"2023-11-06T15:16:08Z"}
{"level":"info","msg":"Server shutdown.","time":"2023-11-06T15:16:08Z"}
{"level":"info","msg":"Stopping bundle loader.","name":"stackable","plugin":"bundle","time":"2023-11-06T15:16:08Z"}

Implementation

Once a server Pod is asked to terminate the following timeline occurs:

  1. The server stops accepting any new queries.

  2. The server waits until all running queries have finished.

  3. If the graceful shutdown doesn’t complete quick enough (e.g. a query runs longer than the graceful shutdown period), after <graceful shutdown period> + 5s safety overhead the Pod gets killed, regardless if it has shut down gracefully or not. This is achieved by setting terminationGracePeriodSeconds on the server Pods. Running queries on the sever will fail.