ADR015: How Should Operators Use Values from ConfigMaps & Secrets

  • Status: accepted

  • Deciders:

    • Felix Hennig

    • Razvan Mihai

    • Sebastian Bernauer

    • Sönke Liebau

    • Natalie Klestrup Röijezon

  • Date: 2022-04-28

Technical Story: #208

Context and Problem Statement

One of the main focus areas of Stackable is making components of the stack work together well. This requires a high degree of composability where one component refers to another compenent in its definition. While this is a simple concept in principle it requires that operators obtain information about the object that is referred to and inject this information into the configuration of the object that refers to it. An example for this is a NiFi cluster definition referring to a ZooKeeper cluster definition via a znode name:

apiVersion: nifi.stackable.tech/v1alpha1
kind: NifiCluster
metadata:
  name: simple-nifi
spec:
  zookeeperConfigMapName: simple-nifi-znode

For this example the NiFi operator needs to somehow insert the content of this config map into specific NiFi config files.

This ADR is about the preferred way of implementing dependencies like described above.

Considered Options

  • Option 1: Inserting ConfigMap/Secret content into target config in the operator

  • Option 2: Mounting ConfigMap/Secret into the pods

Decision Outcome

Chosen option: "Option 2", because it makes better use of Kubernetes features to model dependencies between definitions which in turn will make restart implementations and general debugging easier and more predictable.

This decision will not hold true for all possible scenarios, but should be understood more as a implementation guideline:

If a pod needs to access the content of a ConfigMap or Secret it should do this by mounting the ConfigMap/Secret unless there is a good reason to access the ConfigMap directly in the operator instead.

Positive Consequences

  • Dependencies will be clearly modelled in Kubernetes by mounts, which makes behavior of the entire architecture more predictable

  • Operators need to implement a much simpler logic around watches, as triggering reconciliations is handled via k8s dependencies (mounts)

Negative Consequences

  • Complexity is moved from the operator code into init containers that need to perform insertion of the ConfigMap/Secret content into actual config files

  • Debugging a running configuration becomes harder in cases where users have no access to a shell in the pods, as this would be the only way to actually see the running configuration (#210 has been created to investigate mitigating this)

Pros and Cons of the Options

Option 1 - Inserting ConfigMap/Secret content into target config in the operator

The operator can retrieve the content from ConfigMaps/Secrets during reconciliation and insert the values directly into the target config. The target config would then be written either to a ConfigMap or a Secret, depending on whether sensitive input data is contained in it.

  • Good, because it isolates most of the complexity in the operator

  • Good, because it would allow referencing ConfigMaps and Secrets in different namespaces which would potentially allow for less duplication

  • Good, because it gives an almost complete manifestation of a running config in k8s objects that can be used for debugging

  • Bad, because it would allow referencing ConfigMaps and Secrets in different namespaces, which would dilute namespace separation

  • Bad, because it makes debugging startup issues of products much harder, as pods will simply not appear if required ConfigMaps are missing, effectively bypassing well established k8s patterns (like a pod missing a mount not starting and writing events about why it is not starting)

  • Bad, because the dependencies between pods and configmaps / secrets become very complex and hard to debug as they are not modelled in any way as k8s dependencies (mounts)

  • Bad, because setting up the correct watches in the operator becomes very complex, with a likelyhood of inadvertently forgetting to add a new watch that becomes necessary as CRDs evolve

  • Bad, because it potentially propagates sensitive values to multiple places by copying secret content to additional places

  • Bad, because the Operator is seeing the Secrets. This is not a problem now, but if we could avoid it that would be better

Option 2 - Mounting ConfigMap/Secret into the pods

Instead of accessing the content of ConfigMaps/Secrets in the operator, the operator would simply mount these object into the generated pods. Accessing the content of these mounted objects would then need to be done in an init container, currently this is done with shell commands, but this would be replaced with a more suitable tool in a follow-up ADR.

  • Good, because mounting the ConfigMap/Secret directly into the pod allows the restart controller to automatically restart a Pod if a mounted object changes. Otherwise, changes would not be propagated or the Operator would have to watch the ConfigMap/Secret for changes itself. The watching/restarting is done once by Kubernetes + the restart controller and makes our Operators simpler

  • Bad, because the mounted properties cannot be validated by the Operator. Although Kubernetes at least verifies that all the properties that should be mounted exist

  • Bad, because it is more difficult to see which config is actually used to run the product, as the actual config is only finally assembled inside of the container