Graceful shutdown

You can configure the graceful shutdown as described in Graceful shutdown.

The Druid processes receives a SIGTERM signal when Kubernetes wants to terminate the Pod. It logs the received signal as shown in the log below and initiates a graceful shutdown. After the graceful shutdown timeout runs out, and the process is still running, Kubernetes issues a SIGKILL signal.

Broker

As a default, Druid brokers have 5 minutes to shut down gracefully.

They use the same mechanism described above.

druid 2023-11-13T10:47:13,194 INFO [Thread-55] org.apache.druid.java.util.common.lifecycle.Lifecycle - Lifecycle [module] running shutdown hook
druid 2023-11-13T10:47:13,196 INFO [Thread-55] org.apache.druid.java.util.common.lifecycle.Lifecycle - Stopping lifecycle [module] stage [ANNOUNCEMENTS]
druid 2023-11-13T10:47:13,198 INFO [Thread-55] org.apache.druid.curator.discovery.CuratorServiceAnnouncer - Unannouncing service[DruidNode{serviceName='druid/broker', host='druid-broker-default-0.druid-broker-def
ault.default.svc.cluster.local', bindOnHost=false, port=-1, plaintextPort=-1, enablePlaintextPort=false, tlsPort=8282, enableTlsPort=true}]
druid 2023-11-13T10:47:13,240 INFO [Thread-55] org.apache.druid.curator.announcement.Announcer - Unannouncing [/druid/internal-discovery/BROKER/druid-broker-default-0.druid-broker-default.kuttl-test-striking-poll
iwog.svc.cluster.local:8282]
druid 2023-11-13T10:47:13,246 INFO [NodeRoleWatcher[BROKER]] org.apache.druid.discovery.BaseNodeRoleWatcher - Node [https://druid-broker-default-0.druid-broker-default.default.svc.cluster.loc
al:8282] of role [broker] went offline.
druid 2023-11-13T10:47:13,246 INFO [Thread-55] org.apache.druid.curator.discovery.CuratorDruidNodeAnnouncer - Unannounced self [{"druidNode":{"service":"druid/broker","host":"druid-broker-default-0.druid-broker-d
efault.default.svc.cluster.local","bindOnHost":false,"plaintextPort":-1,"port":-1,"tlsPort":8282,"enablePlaintextPort":false,"enableTlsPort":true},"nodeType":"broker","services":{"lookupNodeService":{"type":"lookupNodeService","lookupTier":"__default"}},"startTime":"2023-11-13T10:41:11.924Z"}].
druid 2023-11-13T10:47:13,247 WARN [CuratorDruidNodeDiscoveryProvider-ListenerExecutor] org.apache.druid.discovery.DruidNodeDiscoveryProvider$ServiceDruidNodeDiscovery - Node[DiscoveryDruidNode{druidNode=DruidNode{serviceName='druid/broker', host='druid-broker-default-0.druid-broker-default.default.svc.cluster.local', bindOnHost=false, port=-1, plaintextPort=-1, enablePlaintextPort=false, tlsPort=8282, enableTlsPort=true}, nodeRole='BROKER', services={lookupNodeService=LookupNodeService{lookupTier='__default'}}', startTime=2023-11-13T10:41:11.924Z}] disappeared but was unknown for service listener [dataNodeService].
druid 2023-11-13T10:47:13,249 INFO [Thread-55] org.apache.druid.java.util.common.lifecycle.Lifecycle - Stopping lifecycle [module] stage [SERVER]
druid 2023-11-13T10:47:13,253 INFO [Thread-55] org.eclipse.jetty.server.AbstractConnector - Stopped ServerConnector@79753f20{SSL, (ssl, http/1.1)}{0.0.0.0:8282}
druid 2023-11-13T10:47:13,253 INFO [Thread-55] org.eclipse.jetty.server.session - node0 Stopped scavenging
druid 2023-11-13T10:47:13,254 INFO [Thread-55] org.eclipse.jetty.server.handler.ContextHandler - Stopped o.e.j.s.ServletContextHandler@3269ae62{/,null,STOPPED}
druid 2023-11-13T10:47:13,259 INFO [Thread-55] org.apache.druid.java.util.common.lifecycle.Lifecycle - Stopping lifecycle [module] stage [NORMAL]
druid 2023-11-13T10:47:13,259 INFO [Thread-55] org.apache.druid.server.coordination.ZkCoordinator - Stopping ZkCoordinator for [DruidServerMetadata{name='druid-broker-default-0.druid-broker-default.default.svc.cluster.local:8282', hostAndPort='null', hostAndTlsPort='druid-broker-default-0.druid-broker-default.default.svc.cluster.local:8282', maxSize=0, tier='_default_tier', type=broker, priority=0}]
druid 2023-11-13T10:47:13,260 INFO [Thread-55] org.apache.druid.server.coordination.SegmentLoadDropHandler - Stopping...
druid 2023-11-13T10:47:13,260 INFO [Thread-55] org.apache.druid.server.coordination.SegmentLoadDropHandler - Stopped.
druid 2023-11-13T10:47:13,260 INFO [Thread-55] org.apache.druid.sql.calcite.schema.MetadataSegmentView - MetadataSegmentView is stopping.
druid 2023-11-13T10:47:13,260 INFO [Thread-55] org.apache.druid.sql.calcite.schema.MetadataSegmentView - MetadataSegmentView Stopped.
druid 2023-11-13T10:47:13,260 INFO [DruidSchema-Cache-0] org.apache.druid.sql.calcite.schema.SegmentMetadataCache - Metadata refresh stopped.
druid 2023-11-13T10:47:13,261 INFO [LookupExtractorFactoryContainerProvider-MainThread] org.apache.druid.query.lookup.LookupReferencesManager - Lookup Management loop exited. Lookup notices are not handled anymore.
druid 2023-11-13T10:47:13,261 INFO [Thread-55] org.apache.druid.guice.LifecycleForkJoinPoolProvider - Shutting down ForkJoinPool [org.apache.druid.guice.LifecycleForkJoinPoolProvider@73741c6e]
druid 2023-11-13T10:47:13,262 INFO [Thread-55] org.apache.druid.client.HttpServerInventoryView - Stopping executor[FilteredHttpServerInventoryView].
druid 2023-11-13T10:47:13,263 INFO [Thread-55] org.apache.druid.client.HttpServerInventoryView - Stopped executor[FilteredHttpServerInventoryView].
druid 2023-11-13T10:47:13,274 ERROR [HttpClient-Netty-Worker-20] com.google.common.util.concurrent.ExecutionList - RuntimeException while executing runnable com.google.common.util.concurrent.Futures$4@6a6cee20 with executor java.util.concurrent.ScheduledThreadPoolExecutor@5bcc07b6[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 6]
druid java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@7f994562[Not completed, task = java.util.concurrent.Executors$RunnableAdapter@2f6e75a9[Wrapped task = com.google.common.util.concurrent.Futures$4@6a6cee20]] rejected from java.util.concurrent.ScheduledThreadPoolExecutor@5bcc07b6[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 6]

Coordinator

As a default, Druid coordinators have 5 minutes to shut down gracefully.

They use the same mechanism described above.

druid 2023-11-13T10:12:30,506 INFO [Thread-54] org.apache.druid.java.util.common.lifecycle.Lifecycle - Lifecycle [module] running shutdown hook
druid 2023-11-13T10:12:30,508 INFO [Thread-54] org.apache.druid.java.util.common.lifecycle.Lifecycle - Stopping lifecycle [module] stage [ANNOUNCEMENTS]
druid 2023-11-13T10:12:30,512 INFO [Thread-54] org.apache.druid.curator.announcement.Announcer - Unannouncing [/druid/internal-discovery/OVERLORD/druid-coordinator-default-0.druid-coordinator-default. default.svc.cluster.local:8281]
druid 2023-11-13T10:12:30,522 INFO [NodeRoleWatcher[OVERLORD]] org.apache.druid.discovery.BaseNodeRoleWatcher - Node [https://druid-coordinator-default-0.druid-coordinator-default. default.svc.cluster.local:8281] of role [overlord] went offline.
druid 2023-11-13T10:12:30,522 INFO [Thread-54] org.apache.druid.curator.discovery.CuratorDruidNodeAnnouncer - Unannounced self [{"druidNode":{"service":"druid/coordinator","host":"druid-coordinator-default-0.druid-coordinator-default. default.svc.cluster.local","bindOnHost":false,"plaintextPort":-1,"port":-1,"tlsPort":8281,"enablePlaintextPort":false,"enableTlsPort":true},"nodeType":"overlord","services":{},"startTime":"2023-11-13T09:54:04.002Z"}].
druid 2023-11-13T10:12:30,523 INFO [Thread-54] org.apache.druid.curator.announcement.Announcer - Unannouncing [/druid/internal-discovery/COORDINATOR/druid-coordinator-default-0.druid-coordinator-default. default.svc.cluster.local:8281]
druid 2023-11-13T10:12:30,524 INFO [Thread-54] org.apache.druid.curator.discovery.CuratorDruidNodeAnnouncer - Unannounced self [{"druidNode":{"service":"druid/coordinator","host":"druid-coordinator-default-0.druid-coordinator-default. default.svc.cluster.local","bindOnHost":false,"plaintextPort":-1,"port":-1,"tlsPort":8281,"enablePlaintextPort":false,"enableTlsPort":true},"nodeType":"coordinator","services":{},"startTime":"2023-11-13T09:54:04.002Z"}].
druid 2023-11-13T10:12:30,600 INFO [Thread-54] org.apache.druid.java.util.common.lifecycle.Lifecycle - Stopping lifecycle [module] stage [SERVER]
druid 2023-11-13T10:12:30,604 INFO [Thread-54] org.eclipse.jetty.server.AbstractConnector - Stopped ServerConnector@2ea2f965{SSL, (ssl, http/1.1)}{0.0.0.0:8281}
druid 2023-11-13T10:12:30,605 INFO [Thread-54] org.eclipse.jetty.server.session - node0 Stopped scavenging
druid 2023-11-13T10:12:30,606 INFO [Thread-54] org.eclipse.jetty.server.handler.ContextHandler - Stopped o.e.j.s.ServletContextHandler@26e588b7{/,jar:file:/stackable/apache-druid-27.0.0/lib/web-console-27.0.0.jar!/org/apache/druid/console,STOPPED}
druid 2023-11-13T10:12:30,610 INFO [Thread-54] org.apache.druid.java.util.common.lifecycle.Lifecycle - Stopping lifecycle [module] stage [NORMAL]
druid 2023-11-13T10:12:30,610 INFO [Thread-54] org.apache.druid.java.util.common.lifecycle.Lifecycle - Stopping lifecycle [task-master] stage [ANNOUNCEMENTS]
druid 2023-11-13T10:12:30,610 INFO [Thread-54] org.apache.druid.java.util.common.lifecycle.Lifecycle - Stopping lifecycle [task-master] stage [SERVER]
druid 2023-11-13T10:12:30,610 INFO [Thread-54] org.apache.druid.java.util.common.lifecycle.Lifecycle - Stopping lifecycle [task-master] stage [NORMAL]
druid 2023-11-13T10:12:30,610 INFO [Thread-54] org.apache.druid.curator.discovery.CuratorServiceAnnouncer - Unannouncing service[DruidNode{serviceName='druid/overlord', host='druid-coordinator-default-0.druid-coordinator-default. default.svc.cluster.local', bindOnHost=false, port=-1, plaintextPort=-1, enablePlaintextPort=false, tlsPort=8281, enableTlsPort=true}]
druid 2023-11-13T10:12:30,702 INFO [Thread-54] org.apache.druid.indexing.common.actions.SegmentAllocationQueue - Not leader anymore. Stopping queue processing.
druid 2023-11-13T10:12:30,703 INFO [Thread-54] org.apache.druid.indexing.overlord.duty.OverlordDutyExecutor - Stopping OverlordDutyExecutor.
druid 2023-11-13T10:12:30,703 INFO [Thread-54] org.apache.druid.indexing.overlord.duty.OverlordDutyExecutor - OverlordDutyExecutor has been stopped.
druid 2023-11-13T10:12:30,703 INFO [Thread-54] org.apache.druid.indexing.overlord.supervisor.SupervisorManager - SupervisorManager stopped.
druid 2023-11-13T10:12:30,703 INFO [TaskQueue-Manager] org.apache.druid.indexing.overlord.TaskQueue - Interrupted, exiting!
druid 2023-11-13T10:12:30,703 INFO [Thread-54] org.apache.druid.indexing.overlord.RemoteTaskRunner - Stopping RemoteTaskRunner...
druid 2023-11-13T10:12:30,703 INFO [Thread-54] org.apache.druid.java.util.common.lifecycle.Lifecycle - Stopping lifecycle [task-master] stage [INIT]
druid 2023-11-13T10:12:30,705 INFO [Thread-54] org.apache.druid.indexing.common.actions.SegmentAllocationQueue - Tearing down segment allocation queue.
druid 2023-11-13T10:12:30,802 INFO [Thread-54] org.apache.druid.client.HttpServerInventoryView - Stopping executor[HttpServerInventoryView].
druid 2023-11-13T10:12:30,802 INFO [Thread-54] org.apache.druid.client.HttpServerInventoryView - Stopped executor[HttpServerInventoryView].
druid 2023-11-13T10:12:30,802 INFO [Thread-54] org.apache.druid.security.basic.authorization.db.updater.CoordinatorBasicAuthorizerMetadataStorageUpdater - CoordinatorBasicAuthorizerMetadataStorageUpdater is stopping.
druid 2023-11-13T10:12:30,802 INFO [Thread-54] org.apache.druid.security.basic.authorization.db.updater.CoordinatorBasicAuthorizerMetadataStorageUpdater - CoordinatorBasicAuthorizerMetadataStorageUpdater is stopped.
druid 2023-11-13T10:12:30,802 INFO [Thread-54] org.apache.druid.security.basic.authentication.db.updater.CoordinatorBasicAuthenticatorMetadataStorageUpdater - CoordinatorBasicAuthenticatorMetadataStorageUpdater is stopping.
druid 2023-11-13T10:12:30,802 INFO [Thread-54] org.apache.druid.security.basic.authentication.db.updater.CoordinatorBasicAuthenticatorMetadataStorageUpdater - CoordinatorBasicAuthenticatorMetadataStorageUpdater is stopped.

Historical

As a default, Druid historicals have 5 minutes to shut down gracefully.

They use the same mechanism described above.

druid 2023-11-13T10:56:54,057 INFO [Thread-55] org.apache.druid.java.util.common.lifecycle.Lifecycle - Lifecycle [module] running shutdown hook
druid 2023-11-13T10:56:54,059 INFO [Thread-55] org.apache.druid.java.util.common.lifecycle.Lifecycle - Stopping lifecycle [module] stage [ANNOUNCEMENTS]
druid 2023-11-13T10:56:54,062 INFO [Thread-55] org.apache.druid.curator.announcement.Announcer - Unannouncing [/druid/internal-discovery/HISTORICAL/druid-historical-default-0.druid-historical-default.default.svc.cluster.local:8283]
druid 2023-11-13T10:56:54,072 INFO [Thread-55] org.apache.druid.curator.discovery.CuratorDruidNodeAnnouncer - Unannounced self [{"druidNode":{"service":"druid/historical","host":"druid-historical-default-0.druid-historical-default.default.svc.cluster.local","bindOnHost":false,"plaintextPort":-1,"port":-1,"tlsPort":8283,"enablePlaintextPort":false,"enableTlsPort":true},"nodeType":"historical","services":{"dataNodeService":{"type":"dataNodeService","tier":"_default_tier","maxSize":1000000000,"type":"historical","serverType":"historical","priority":0},"lookupNodeService":{"type":"lookupNodeService","lookupTier":"__default"}},"startTime":"2023-11-13T10:41:10.024Z"}].
druid 2023-11-13T10:56:54,081 INFO [Thread-55] org.apache.druid.curator.announcement.Announcer - Unannouncing [/druid/segments/druid-historical-default-0.druid-historical-default.default.svc.cluster.local:8283/druid-historical-default-0.druid-historical-default.default.svc.cluster.local:8283_historical__default_tier_2023-11-13T10:42:12.401Z_07a7108a06df494b8f8d7c01c841384a0]
druid 2023-11-13T10:56:54,083 INFO [Thread-55] org.apache.druid.curator.announcement.Announcer - Unannouncing [/druid/announcements/druid-historical-default-0.druid-historical-default.default.svc.cluster.local:8283]
druid 2023-11-13T10:56:54,084 INFO [Thread-55] org.apache.druid.java.util.common.lifecycle.Lifecycle - Stopping lifecycle [module] stage [SERVER]
druid 2023-11-13T10:56:54,089 INFO [Thread-55] org.eclipse.jetty.server.AbstractConnector - Stopped ServerConnector@1a632663{SSL, (ssl, http/1.1)}{0.0.0.0:8283}
druid 2023-11-13T10:56:54,089 INFO [Thread-55] org.eclipse.jetty.server.session - node0 Stopped scavenging
druid 2023-11-13T10:56:54,090 INFO [Thread-55] org.eclipse.jetty.server.handler.ContextHandler - Stopped o.e.j.s.ServletContextHandler@3b9d85c2{/,null,STOPPED}
druid 2023-11-13T10:56:54,093 INFO [Thread-55] org.apache.druid.java.util.common.lifecycle.Lifecycle - Stopping lifecycle [module] stage [NORMAL]
druid 2023-11-13T10:56:54,094 INFO [Thread-55] org.apache.druid.server.coordination.ZkCoordinator - Stopping ZkCoordinator for [DruidServerMetadata{name='druid-historical-default-0.druid-historical-default.default.svc.cluster.local:8283', hostAndPort='null', hostAndTlsPort='druid-historical-default-0.druid-historical-default.default.svc.cluster.local:8283', maxSize=1000000000, tier='_default_tier', type=historical, priority=0}]
druid 2023-11-13T10:56:54,094 INFO [Thread-55] org.apache.druid.server.coordination.SegmentLoadDropHandler - Stopping...
druid 2023-11-13T10:56:54,094 INFO [Thread-55] org.apache.druid.server.coordination.SegmentLoadDropHandler - Stopped.
druid 2023-11-13T10:56:54,094 INFO [LookupExtractorFactoryContainerProvider-MainThread] org.apache.druid.query.lookup.LookupReferencesManager - Lookup Management loop exited. Lookup notices are not handled anymore.
druid 2023-11-13T10:56:54,096 INFO [Thread-55] org.apache.druid.security.basic.authorization.db.cache.CoordinatorPollingBasicAuthorizerCacheManager - CoordinatorPollingBasicAuthorizerCacheManager is stopping.
druid 2023-11-13T10:56:54,096 INFO [Thread-55] org.apache.druid.security.basic.authorization.db.cache.CoordinatorPollingBasicAuthorizerCacheManager - CoordinatorPollingBasicAuthorizerCacheManager is stopped.
druid 2023-11-13T10:56:54,096 INFO [Thread-55] org.apache.druid.security.basic.authentication.db.cache.CoordinatorPollingBasicAuthenticatorCacheManager - CoordinatorPollingBasicAuthenticatorCacheManager is stopping.
druid 2023-11-13T10:56:54,096 INFO [Thread-55] org.apache.druid.security.basic.authentication.db.cache.CoordinatorPollingBasicAuthenticatorCacheManager - CoordinatorPollingBasicAuthenticatorCacheManager is stopped.
druid 2023-11-13T10:56:54,108 INFO [Curator-Framework-0] org.apache.curator.framework.imps.CuratorFrameworkImpl - backgroundOperationsLoop exiting
druid 2023-11-13T10:56:54,212 INFO [Thread-55] org.apache.zookeeper.ZooKeeper - Session: 0x10011760e0e0007 closed
druid 2023-11-13T10:56:54,212 INFO [main-EventThread] org.apache.zookeeper.ClientCnxn - EventThread shut down for session: 0x10011760e0e0007
druid 2023-11-13T10:56:54,212 INFO [Thread-55] org.apache.druid.java.util.common.lifecycle.Lifecycle - Stopping lifecycle [module] stage [INIT]

MiddleManager

As a default, Druid middle managers have 5 minutes to shut down gracefully.

The middle manager can be terminated gracefully by disabling it. Meaning the overlord will not send any new tasks and the middle manager terminates after all tasks are finished or the termination grace period is exceeded.

druid 2023-11-13T10:42:50,569 INFO [stackable_lifecycle_pre_stop] - Disable middle manager to stop overlord from sending tasks: {"druid-middlemanager-default-0.druid-middlemanager-default.default.svc.cluster.local:8291":"disabled"}
druid 2023-11-13T10:42:50,578 INFO [stackable_lifecycle_pre_stop] - Check if termination grace period (300 seconds) is reached...
druid 2023-11-13T10:42:50,586 INFO [stackable_lifecycle_pre_stop] - Check if all tasks are finished... Running: []
druid 2023-11-13T10:42:50,591 INFO [stackable_lifecycle_pre_stop] - All tasks finished!
druid 2023-11-13T10:42:50,605 INFO [stackable_lifecycle_pre_stop] - All done!
druid ++ handle_term_signal
druid ++ '[' 11 ']'
druid ++ kill -TERM 11
druid + trap - TERM
druid + wait 11
druid 2023-11-13T10:42:50,652 INFO [Thread-52] org.apache.druid.java.util.common.lifecycle.Lifecycle - Lifecycle [module] running shutdown hook
druid 2023-11-13T10:42:50,654 INFO [Thread-52] org.apache.druid.java.util.common.lifecycle.Lifecycle - Stopping lifecycle [module] stage [ANNOUNCEMENTS]
druid 2023-11-13T10:42:50,656 INFO [Thread-52] org.apache.druid.curator.announcement.Announcer - Unannouncing [/druid/internal-discovery/MIDDLE_MANAGER/druid-middlemanager-default-0.druid-middlemanager-default.default.svc.cluster.local:8291]
druid 2023-11-13T10:42:50,671 INFO [Thread-52] org.apache.druid.curator.discovery.CuratorDruidNodeAnnouncer - Unannounced self [{"druidNode":{"service":"druid/middlemanager","host":"druid-middlemanager-default-0.druid-middlemanager-default.default.svc.cluster.local","bindOnHost":false,"plaintextPort":-1,"port":-1,"tlsPort":8291,"enablePlaintextPort":false,"enableTlsPort":true},"nodeType":"middleManager","services":{"workerNodeService":{"type":"workerNodeService","ip":"druid-middlemanager-default-0.druid-middlemanager-default.default.svc.cluster.local","capacity":1,"version":"0","category":"_default_worker_category"}},"startTime":"2023-11-13T10:41:10.341Z"}].
druid 2023-11-13T10:42:50,675 INFO [Thread-52] org.apache.druid.java.util.common.lifecycle.Lifecycle - Stopping lifecycle [module] stage [SERVER]
druid 2023-11-13T10:42:50,677 INFO [Thread-52] org.eclipse.jetty.server.AbstractConnector - Stopped ServerConnector@2f51b100{SSL, (ssl, http/1.1)}{0.0.0.0:8291}
druid 2023-11-13T10:42:50,677 INFO [Thread-52] org.eclipse.jetty.server.session - node0 Stopped scavenging
druid 2023-11-13T10:42:50,679 INFO [Thread-52] org.eclipse.jetty.server.handler.ContextHandler - Stopped o.e.j.s.ServletContextHandler@28705150{/,null,STOPPED}
druid 2023-11-13T10:42:50,683 INFO [Thread-52] org.apache.druid.java.util.common.lifecycle.Lifecycle - Stopping lifecycle [module] stage [NORMAL]
druid 2023-11-13T10:42:50,684 INFO [Thread-52] org.apache.druid.indexing.overlord.ForkingTaskRunner - Waiting up to 300,000ms for shutdown.
druid 2023-11-13T10:42:50,685 INFO [Thread-52] org.apache.druid.indexing.overlord.ForkingTaskRunner - Finished stopping in 2ms.
druid 2023-11-13T10:42:50,685 INFO [Thread-52] org.apache.druid.indexing.worker.WorkerCuratorCoordinator - Stopping WorkerCuratorCoordinator for worker[druid-middlemanager-default-0.druid-middlemanager-default.default.svc.cluster.local:8291]
druid 2023-11-13T10:42:50,686 INFO [Thread-52] org.apache.druid.curator.announcement.Announcer - Unannouncing [/druid/indexer/announcements/druid-middlemanager-default-0.druid-middlemanager-default.default.svc.cluster.local:8291]
druid 2023-11-13T10:42:50,688 INFO [Thread-52] org.apache.druid.indexing.overlord.ForkingTaskRunner - Waiting up to 300,000ms for shutdown.
druid 2023-11-13T10:42:50,688 INFO [Thread-52] org.apache.druid.indexing.overlord.ForkingTaskRunner - Finished stopping in 0ms.
druid 2023-11-13T10:42:50,688 INFO [Thread-52] org.apache.druid.security.basic.authorization.db.cache.CoordinatorPollingBasicAuthorizerCacheManager - CoordinatorPollingBasicAuthorizerCacheManager is stopping.
druid 2023-11-13T10:42:50,689 INFO [Thread-52] org.apache.druid.security.basic.authorization.db.cache.CoordinatorPollingBasicAuthorizerCacheManager - CoordinatorPollingBasicAuthorizerCacheManager is stopped.
druid 2023-11-13T10:42:50,689 INFO [Thread-52] org.apache.druid.security.basic.authentication.db.cache.CoordinatorPollingBasicAuthenticatorCacheManager - CoordinatorPollingBasicAuthenticatorCacheManager is stopping.
druid 2023-11-13T10:42:50,689 INFO [Thread-52] org.apache.druid.security.basic.authentication.db.cache.CoordinatorPollingBasicAuthenticatorCacheManager - CoordinatorPollingBasicAuthenticatorCacheManager is stopped.
druid 2023-11-13T10:42:50,704 INFO [Curator-Framework-0] org.apache.curator.framework.imps.CuratorFrameworkImpl - backgroundOperationsLoop exiting
druid 2023-11-13T10:42:50,808 INFO [Thread-52] org.apache.zookeeper.ZooKeeper - Session: 0x10011760e0e0008 closed
druid 2023-11-13T10:42:50,808 INFO [main-EventThread] org.apache.zookeeper.ClientCnxn - EventThread shut down for session: 0x10011760e0e0008
druid 2023-11-13T10:42:50,808 INFO [Thread-52] org.apache.druid.java.util.common.lifecycle.Lifecycle - Stopping lifecycle [module] stage [INIT]

Router

As a default, Druid routers have 5 minutes to shut down gracefully.

They use the same mechanism described above.

druid 2023-11-13T10:53:13,401 INFO [Thread-70] org.apache.druid.java.util.common.lifecycle.Lifecycle - Lifecycle [module] running shutdown hook
druid 2023-11-13T10:53:13,403 INFO [Thread-70] org.apache.druid.java.util.common.lifecycle.Lifecycle - Stopping lifecycle [module] stage [ANNOUNCEMENTS]
druid 2023-11-13T10:53:13,406 INFO [Thread-70] org.apache.druid.curator.announcement.Announcer - Unannouncing [/druid/internal-discovery/ROUTER/druid-router-default-0.druid-router-default.default.svc.cluster.local:9088]
druid 2023-11-13T10:53:13,501 INFO [Thread-70] org.apache.druid.curator.discovery.CuratorDruidNodeAnnouncer - Unannounced self [{"druidNode":{"service":"druid/router","host":"druid-router-default-0.druid-router-default.default.svc.cluster.local","bindOnHost":false,"plaintextPort":-1,"port":-1,"tlsPort":9088,"enablePlaintextPort":false,"enableTlsPort":true},"nodeType":"router","services":{},"startTime":"2023-11-13T10:41:23.188Z"}].
druid 2023-11-13T10:53:13,501 INFO [Thread-70] org.apache.druid.curator.discovery.CuratorServiceAnnouncer - Unannouncing service[DruidNode{serviceName='druid/router', host='druid-router-default-0.druid-router-default.default.svc.cluster.local', bindOnHost=false, port=-1, plaintextPort=-1, enablePlaintextPort=false, tlsPort=9088, enableTlsPort=true}]
druid 2023-11-13T10:53:13,587 INFO [Thread-70] org.apache.druid.java.util.common.lifecycle.Lifecycle - Stopping lifecycle [module] stage [SERVER]
druid 2023-11-13T10:53:13,591 INFO [Thread-70] org.eclipse.jetty.server.AbstractConnector - Stopped ServerConnector@77732366{SSL, (ssl, http/1.1)}{0.0.0.0:9088}
druid 2023-11-13T10:53:13,591 INFO [Thread-70] org.eclipse.jetty.server.session - node0 Stopped scavenging
druid 2023-11-13T10:53:13,596 INFO [Thread-70] org.eclipse.jetty.server.handler.ContextHandler - Stopped o.e.j.s.ServletContextHandler@487f025{/,jar:file:/stackable/apache-druid-27.0.0/lib/web-console-27.0.0.jar!/org/apache/druid/console,STOPPED}
druid 2023-11-13T10:53:13,687 INFO [Thread-70] org.apache.druid.java.util.common.lifecycle.Lifecycle - Stopping lifecycle [module] stage [NORMAL]
druid 2023-11-13T10:53:13,688 INFO [Thread-70] org.apache.druid.security.basic.authorization.db.cache.CoordinatorPollingBasicAuthorizerCacheManager - CoordinatorPollingBasicAuthorizerCacheManager is stopping.
druid 2023-11-13T10:53:13,688 INFO [Thread-70] org.apache.druid.security.basic.authorization.db.cache.CoordinatorPollingBasicAuthorizerCacheManager - CoordinatorPollingBasicAuthorizerCacheManager is stopped.
druid 2023-11-13T10:53:13,688 INFO [Thread-70] org.apache.druid.security.basic.authentication.db.cache.CoordinatorPollingBasicAuthenticatorCacheManager - CoordinatorPollingBasicAuthenticatorCacheManager is stopping.
druid 2023-11-13T10:53:13,688 INFO [Thread-70] org.apache.druid.security.basic.authentication.db.cache.CoordinatorPollingBasicAuthenticatorCacheManager - CoordinatorPollingBasicAuthenticatorCacheManager is stopped.
druid 2023-11-13T10:53:13,790 INFO [Curator-Framework-0] org.apache.curator.framework.imps.CuratorFrameworkImpl - backgroundOperationsLoop exiting
druid 2023-11-13T10:53:13,895 INFO [Thread-70] org.apache.zookeeper.ZooKeeper - Session: 0x10011760e0e000a closed
druid 2023-11-13T10:53:13,895 INFO [main-EventThread] org.apache.zookeeper.ClientCnxn - EventThread shut down for session: 0x10011760e0e000a
druid 2023-11-13T10:53:13,895 INFO [Thread-70] org.apache.druid.java.util.common.lifecycle.Lifecycle - Stopping lifecycle [module] stage [INIT]