`SecretClass`

A SecretClass is a cluster-global Kubernetes resource that defines a category of secrets that the Secret Operator knows how to provision.

This is intended to provide an abstraction between how the secret is used ("I need a certificate for my cluster’s TLS PKI") and how it is provisioned (automatically and generated by the operator’s internal CA, provisioned by the cluster administrator, or provisioned by an external service such as Hashicorp Vault).

A SecretClass looks like this:

---
apiVersion: secrets.stackable.tech/v1alpha1
kind: SecretClass
metadata:
  name: tls
spec:
  backend: (1)
    autoTls: (2)
      ca:
        secret:
          name: secret-provisioner-tls-ca
          namespace: default
        autoGenerate: true
    # or... (1)
    k8sSearch: (3)
      searchNamespace:
        pod: {}
        # or...
        name: my-namespace

1	Backends are mutually exclusive, only one may be used by each `SecretClass`
2	Configures and selects the `autoTls` backend
3	Configures and selects the `k8sSearch` backend

Backend

Each SecretClass is a associated with a single backend, which dictates the mechanism for issuing that kind of secret.

`autoTls`

Format: TLS (PEM)

Issues a TLS certificate signed by the Secret Operator. The certificate authority can be provided by the administrator, or managed automatically by the Secret Operator.

A new certificate and keypair will be generated and signed for each Pod, keys or certificates are never reused.

Attributes of the certificate (such as the expiration date, fingerprint, or serial number) will be regenerated for each Pod, and should not be expected to be stable.

Scopes are used to populate the claims (such as subjectAlternateName) of the provisioned certificates.

Certificate lifetime

We generally aim to use as short-lived certificates as possible. Short lifetime periods require frequent restarts of services, which may be totally unnoticeable for some products, annoying for others or fatal for products with a single point of failure and no recovery from outages (e.g. Trino coordinator).

We have spent a considerate amount of time thinking about this issue and decided on the following compromises:

To enforce security constraints, cluster administrators can set a maximum allowed certificate lifetime on the SecretClass issuing the certificates (defaults to 15d).
We default to a certificate lifetime of 24h.
Pods consuming the certificate can request a longer lifetime or shutdown expiration buffer via annotations on the Volume. If they request a longer lifetime than the SecretClass allows, it will be shortened to the maximum allowed lifetime.
To avoid stampeding herds during restarts and spread out the load, certificate durations are lowered by up to 20%.
The Pods will be evicted 6h before the certificate expires, to ensure that no Pods are left running with expired secrets. Consumers can override this buffer using Volume annotations. This buffer must be long enough that the product is guaranteed to gracefully shut down.

Most of our product operators will not set any specific certificate lifetime, so the default applies. In case an operator sets a higher lifetime, a tracking issue must be created to document and track the steps to reduce the certificate lifetime.

Users can use podOverrides to extend the certificate lifetime by adding volume annotations. We might add native support for customizing certificate lifetimes in the future to the Stacklet CRDs.

Certificate Authority rotation

Certificate authorities also have a limited lifetime, and need to be rotated before they expire to avoid cluster disruption.

If configured to provision its own CA (autoTls.ca.autoGenerate), the Secret Operator will create CA certificates that are valid for 2 years (autoTls.ca.caCertificateLifetime), and initiate rotation once less than half of that time remains.

To avoid disruption and let the new CA propagate through the cluster, the Secret Operator will prefer using the oldest CA that will last for the entire lifetime of the issued certificate.

Expired CA certificates will currently not be deleted automatically. They should be cleaned up manually.

Reference

spec:
  backend:
    autoTls:
      ca:
        secret:
          name: secret-provisioner-tls-ca
          namespace: default
        autoGenerate: true
        caCertificateLifetime: 700d
      maxCertificateLifetime: 15d # optional

autoTls: Declares that the autoTls backend is used.
autoTls.ca: Configures the certificate authority used to issue Pod certificates.
autoTls.ca.secret: Reference (name and namespace) to a K8s Secret object where the CA certificate and key is stored in the keys ca.crt and ca.key respectively.
autoTls.ca.autoGenerate: Whether the certificate authority should be provisioned and managed by the Secret Operator.
autoTls.ca.caCertificateLifetime: The lifetime of the certificate authority’s root certificate.
autoTls.maxCertificateLifetime: Maximum lifetime the created certificates are allowed to have. In case consumers request a longer lifetime than allowed by this setting, the lifetime will be the minimum of both.

`kerberosKeytab`

Format: Kerberos

Creates a Kerberos keytab file for a selected realm. The Kerberos KDC and administrator credentials must be provided by the administrator.

Only MIT Kerberos (krb5) and Active Directory are currently supported. Heimdal is not supported.

Principals will be created dynamically if they do not already exist.

The administrator keytab must have permission to add principals and get their keys. This corresponds to the flags ae in kadm5.acl.

Active Directory

Principal Conflicts

We recommend that each Active Directory domain should only be used by a single Kubernetes cluster.

This is because each pod, service, and node may be provisioned a principal matching its hostname, and principal names must be unique within a single AD domain. The Stackable Secret Operator will cache and reuse these credentials within a single Kubernetes cluster, but will not share them across multiple clusters.

If the same AD domain is shared between multiple Kubernetes clusters, the following must be unique across the AD domain:

The Kubernetes Nodes' names and fully qualified domain names
The Kubernetes Namespaces' names (only Namespaces that use Kerberos)

Reference

spec:
  backend:
    kerberosKeytab:
      realmName: CLUSTER.LOCAL
      kdc: krb5-kdc
      admin:
        mit:
          kadminServer: krb5-kdc
        # or...
        activeDirectory:
          # ldapServer must match the AD Domain Controller's FQDN or GSSAPI authn will fail
          # You may need to set AD as your fallback DNS resolver in your Kube DNS Corefile
          ldapServer: addc.example.com
          ldapTlsCaSecret:
            namespace: default
            name: secret-operator-ad-ca
          passwordCacheSecret:
            namespace: default
            name: secret-operator-ad-passwords
          userDistinguishedName: CN=Users,DC=sble,DC=test
          schemaDistinguishedName: CN=Schema,CN=Configuration,DC=sble,DC=test
      adminKeytabSecret:
        namespace: default
        name: secret-provisioner-keytab
      adminPrincipal: stackable-secret-operator

kerberosKeytab: Declares that the kerberosKeytab backend is used.
kerberosKeytab.realmName: The name of the Kerberos realm. This should be provided by the Kerberos administrator.
kerberosKeytab.kdc: The hostname of the Kerberos Key Distribution Center (KDC). This should be provided by the Kerberos administrator.
kerberosKeytab.admin.mit: Credentials should be provisioned in a MIT Kerberos Admin Server.
kerberosKeytab.admin.mit.kadminServer: The hostname of the Kerberos Admin Server. This should be provided by the Kerberos administrator.
kerberosKeytab.admin.activeDirectory: Credentials should be provisioned in a Microsoft Active Directory domain.
kerberosKeytab.admin.activeDirectory.ldapServer: An AD LDAP server, such as the AD Domain Controller. This must match the server’s FQDN, or GSSAPI authentication will fail.
kerberosKeytab.admin.activeDirectory.ldapTlsCaSecret: Reference (name and namespace) to a K8s Secret object containing the TLS CA (in ca.crt) that the LDAP server’s certificate should be authenticated against.
kerberosKeytab.admin.activeDirectory.passwordCacheSecret: Reference (name and namespace) to a K8s Secret object where workload passwords will be stored. This must not be accessible to end users.
kerberosKeytab.admin.activeDirectory.userDistinguishedName: The root Distinguished Name (DN) where service accounts should be provisioned, typically CN=Users,{domain_dn}.
kerberosKeytab.admin.activeDirectory.schemaDistinguishedName: The root Distinguished Name (DN) for AD-managed schemas, typically CN=Schema,CN=Configuration,{domain_dn}.
kerberosKeytab.adminKeytabSecret: Reference (name and namespace) to a K8s Secret object where a keytab with administrative privileges is stored in the key keytab.
kerberosKeytab.adminPrincipal: The name of the Kerberos principal to be used by the Secret Operator. This should be provided by the Kerberos administrator. The credentials for this principal must be stored in the keytab (adminKeytabSecret).

`k8sSearch`

Format: Free-form

This backend can be used to mount Secret across namespaces into pods. The Secret object is selected based on two things:

The scopes specified on the Volume using the attribute secrets.stackable.tech/scope.
The label secrets.stackable.tech/class specified in the Secret it’s self. This must match the name of the SecretClass.

Each field in this Secret is mapped to one file. It is suggested these Secret objects should follow one of the formats defined in this document.

In the example below, given the three object definitions for a Pod, a SecretClass and a Secret, the operator will first read the Pod’s volume attributes then look up the secret class. The k8sSearch backend will look up the Secret object labeled with secrets.stackable.tech/class: admin-credentials-class (the name of the secret class) and mount the fields of the Secret as files into the container at the specified mount point (/credentials). Please note that the contents in the volume will not update when the Secret content changes. A Pod restart is needed to refresh the Secret contents on disk.

apiVersion: apps/v1
kind: Pod
metadata:
  name: my-app
  labels:
    app.kubernetes.io/name: my-app
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: my-app
  containers:
    #... skipped for brevity
    volumeMounts:
    - name: admin-credentials-volume
      mountPath: /credentials
  volumes:
  - name: admin-credentials-volume
    ephemeral:
      volumeClaimTemplate:
        metadata:
          annotations:
            secrets.stackable.tech/class: admin-credentials-class
            secrets.stackable.tech/scope: pod
        spec:
          storageClassName: secrets.stackable.tech
          accessModes:
            - ReadWriteOnce
          resources:
            requests:
              storage: "1"
---
apiVersion: secrets.stackable.tech/v1alpha1
kind: SecretClass
metadata:
  name: admin-credentials-class
spec:
  backend:
   k8sSearch:
      searchNamespace:
        pod: {}
---
apiVersion: v1
kind: Secret
metadata:
  name: my-admin-credentials
  labels:
    secrets.stackable.tech/class: admin-credentials-class
stringData:
  user: admin
  password: secret

Scopes are translated into additional label filters of the form secrets.stackable.tech/$SCOPE: $SCOPE_VALUE. For example, a Pod named foo mounting a k8sSearch secret with the pod scope would add the label filter secrets.stackable.tech/pod: foo.

Reference

spec:
  backend:
    k8sSearch:
      searchNamespace:
        pod: {}
        # or...
        name: my-namespace

k8sSearch: Declares that the k8sSearch backend is used.
k8sSearch.searchNamespace: Configures the namespace searched for Secret objects.
k8sSearch.searchNamespace.pod: The Secret objects are located in the same namespace as the Pod object. Should be used for secrets that are provisioned by the application administrator.
k8sSearch.searchNamespace.name: The Secret objects are located in a single global namespace. Should be used for secrets that are provisioned by the cluster administrator.

Format

The k8sSearch backend doesn’t use a particular format, since the underlying Secret may contain any kind of free-form data. If a specific format is requested then the source format of the secret is inferred (based on the keys present in the secret), and a conversion is performed if required.

If a format is requested then only the keys relevant to the format will be mounted. For example, given a secret containing the files ca.crt, tls.crt, tls.key, and tls.config, the tls.config key will be projected to a file if no format is requested, but not if the TLS (PEM) format is requested.

Format

A format describes a set of artifacts (files and their respective contents) produced by a backend.

Each backend should conform to at least one common format. This is intended to allow cluster operators to switch between interoperable backends with minimal impact on secret consumers.

TLS (PEM)

Name: tls-pem

The secret contains the following files:

ca.crt: The certificate of the Certificate Authority (and associated chain) that has signed the certificate, in the PEM format.
tls.crt: The certificate identifying the Pod, in the PEM format.
tls.key: The private key corresponding to tls.crt, in the PEM format.

A TLS secret can also be converted into TLS (PKCS#12).

TLS (PKCS#12)

Name: tls-pkcs12

The secret contains the following files:

keystore.p12: A bundle of the private key and certificate (including the CA certificates), in the PKCS#12 format.
truststore.p12: A bundle of trusted CA certificates, in the PKCS#12 format.

Both stores are encrypted, with an empty string as the passphrase.

When using the k8sSearch backend, it is strongly recommended to store secrets in the use the TLS (PEM) format instead. The secret operator supports converting PEM keypairs into PKCS#12, but not the other way around.

Kerberos

Name: kerberos

The secret contains the following files:

krb5.conf: Kerberos configuration file for authenticating against the Kerberos realm.
keytab: A Kerberos keytab file containing credentials for all requested principals.