Stackable operators offer different cluster operations to control the reconciliation process. This is useful when updating operators, debugging or testing of new settings:
reconciliationPaused- Stop the operator from reconciling the cluster spec. The status will still be updated.
stopped- Stop all running pods but keep updating all deployed resources like
Servicesand the cluster status.
If not specified,
clusterOperation.stopped default to
--- apiVersion: mycluster.stackable.tech/v1alpha1 kind: MyCluster metadata: name: my-cluster spec: clusterOperation: reconciliationPaused: false (1) stopped: false (2)
Sometimes it is necessary to restart services deployed in Kubernetes. A service restart should induce as little disruption as possible, ideally none.
Most operators create StatefulSet objects for the products they manage and Kubernetes offers a rollout mechanism to restart them. You can use
kubectl rollout restart statefulset to restart a StatefulSet previously created by an operator.
To illustrate how to use the command line to restart one or more Pods, we will assume you used the Stackable HDFS Operator to deploy an HDFS stacklet called
This stacklet will consist, among other things, of three StatefulSets created for each HDFS role:
journalnode. Let’s list them:
❯ kubectl get statefulset -l app.kubernetes.io/instance=dumbo NAME READY AGE dumbo-datanode-default 2/2 4m41s dumbo-journalnode-default 1/1 4m41s dumbo-namenode-default 2/2 4m41s
To restart the HDFS DataNode Pods, run:
❯ kubectl rollout restart statefulset dumbo-datanode-default statefulset.apps/dumbo-datanode-default restarted
Sometimes you want to restart all Pods of a stacklet and not just individual roles. This can be achieved in a similar manner by using labels instead of StatefulSet names. Continuing with the example above, to restart all HDFS Pods you would have to run:
❯ kubectl rollout restart statefulset --selector app.kubernetes.io/instance=dumbo
To wait for all Pods to be running again:
❯ kubectl rollout status statefulset --selector app.kubernetes.io/instance=dumbo
Here we used the label
app.kubernetes.io/instance=dumbo to select all Pods that belong to a specific HDFS stacklet. This label is created by the operator and
dumbo is the name of the HDFS stacklet as specified in the custom resource. You can add more labels to make finer grained restarts.
The Commons Operator of the Stackable Platform may restart Pods automatically, for purposes such as ensuring that TLS certificates are up-to-date. For details, see the Commons Operator documentation.