SecretClass

A SecretClass is a cluster-global Kubernetes resource that defines a category of secrets that the Secret Operator knows how to provision.

This is intended to provide an abstraction between how the secret is used ("I need a certificate for my cluster’s TLS PKI") and how it is provisioned (automatically and generated by the operator’s internal CA, provisioned by the cluster administrator, or provisioned by an external service such as Hashicorp Vault).

A SecretClass looks like this:

---
apiVersion: secrets.stackable.tech/v1alpha1
kind: SecretClass
metadata:
  name: tls
spec:
  backend: (1)
    autoTls: (2)
      ca:
        secret:
          name: secret-provisioner-tls-ca
          namespace: default
        autoGenerate: true
    # or... (1)
    k8sSearch: (3)
      searchNamespace:
        pod: {}
        # or...
        name: my-namespace
1 Backends are mutually exclusive, only one may be used by each SecretClass
2 Configures and selects the autoTls backend
3 Configures and selects the k8sSearch backend

Backend

Each SecretClass is a associated with a single backend, which dictates the mechanism for issuing that kind of secret.

autoTls

Format: TLS (PEM)

Issues a TLS certificate signed by the Secret Operator. The certificate authority can be provided by the administrator, or managed automatically by the Secret Operator.

A new certificate and key pair will be generated and signed for each Pod, keys or certificates are never reused.

Attributes of the certificate (such as the expiration date, fingerprint, or serial number) will be regenerated for each Pod, and should not be expected to be stable.

Scopes are used to populate the claims (such as subjectAlternateName) of the provisioned certificates.

TLS certificate key pair generation

Currently, only RSA is supported in the autoTls backend. You can however configure the key length for generated private keys. If not specified it will default to 2048 bits.

---
apiVersion: secrets.stackable.tech/v1alpha1
kind: SecretClass
metadata:
  name: tls
spec:
  backend:
    autoTls:
      ca:
        secret:
          name: secret-provisioner-tls-ca
          namespace: default
        keyGeneration: (1)
          rsa: (2)
            length: 4096 (3)
1 autoTls.ca.keyGeneration specifies which algorithm and additional parameters are used
2 autoTls.ca.keyGeneration.rsa specifies the RSA key pair algorithm (RSA currently is the only one supported)
3 autoTls.ca.keyGeneration.rsa.length specifies the amount of bits used for generating the RSA key pair. Currently, 2048, 3072 and 4096 are supported. Defaults to 2048 bits.

CAUTION

Using more than 2048 bits will significantly increase the computation time to create new key pairs. The SSL Labs SSL and TLS Deployment Best Practices as of 2024-10-01 recommend

For most websites, using RSA keys stronger than 2,048 bits and ECDSA keys stronger than 256 bits is a waste of CPU power and might impair user experience

If options higher than 2048 are chosen, the CPU resources for the secret operator should be increased in order to avoid Pods being stuck in Pending waiting for the computation of their key pair.

Certificate lifetime

By default the Secret Operator will generally aim to use as short-lived certificates as possible. Short lifetime periods require frequent restarts of services, which may be totally unnoticeable for some products, annoying for others or fatal for products with a single point of failure and no recovery from outages (e.g. Trino coordinator).

This choice was made deliberately after a considerate amount of time was spent thinking about this issue. The following compromises were made:

  1. To enforce security constraints, cluster administrators can set a maximum allowed certificate lifetime on the SecretClass issuing the certificates (defaults to 15d).

  2. The certificate lifetime is 24h.

  3. Pods consuming the certificate can request a longer lifetime or shutdown expiration buffer via annotations on the Volume. If they request a longer lifetime than the SecretClass allows, it will be shortened to the maximum allowed lifetime.

  4. To avoid stampeding herds during restarts and spread out the load, certificate durations are lowered by up to 20%.

  5. The Pods will be evicted 6h before the certificate expires, to ensure that no Pods are left running with expired secrets. Consumers can override this buffer using Volume annotations. This buffer must be long enough that the product is guaranteed to gracefully shut down.

Most Stackable product operators will not set any specific certificate lifetime, so the default applies. In case an operator sets a higher lifetime, a tracking issue must be created to document and track the steps to reduce the certificate lifetime.

Users can use podOverrides to extend the certificate lifetime by adding volume annotations. Native support for customizing certificate lifetimes in Stacklet CRDs might be added in the future.

Certificate Authority rotation

Certificate authorities also have a limited lifetime, and need to be rotated before they expire to avoid cluster disruption.

If configured to provision its own CA (autoTls.ca.autoGenerate), the Secret Operator will create CA certificates that are valid for 2 years (autoTls.ca.caCertificateLifetime), and initiate rotation once less than half of that time remains.

To avoid disruption and let the new CA propagate through the cluster, the Secret Operator will prefer using the oldest CA that will last for the entire lifetime of the issued certificate.

Expired CA certificates will currently not be deleted automatically. They should be cleaned up manually.

Reference

spec:
  backend:
    autoTls:
      ca:
        secret:
          name: secret-provisioner-tls-ca
          namespace: default
        autoGenerate: true
        caCertificateLifetime: 700d
      maxCertificateLifetime: 15d # optional
autoTls

Declares that the autoTls backend is used.

autoTls.ca

Configures the certificate authority used to issue Pod certificates.

autoTls.ca.secret

Reference (name and namespace) to a K8s Secret object where the CA certificate and key is stored in the keys ca.crt and ca.key respectively.

autoTls.ca.autoGenerate

Whether the certificate authority should be provisioned and managed by the Secret Operator.

autoTls.ca.caCertificateLifetime

The lifetime of the certificate authority’s root certificate.

autoTls.maxCertificateLifetime

Maximum lifetime the created certificates are allowed to have. In case consumers request a longer lifetime than allowed by this setting, the lifetime will be the minimum of both.

experimentalCertManager

Format: TLS (PEM)

Injects a TLS certificate issued by Cert-Manager.

This backend is experimental, and subject to change.
This backend requires Cert-Manager to already be installed and configured.

A new certificate will be requested the first time it is used by a Pod, it will be reused after that (subject to Cert-Manager’s renewal rules).

Node-scoped requests will cause a Pod to become "sticky" to the Node that it was first scheduled to (like k8sSearch, but unlike autoTls).

Reference

spec:
  backend:
    experimentalCertManager:
      issuer:
        kind: Issuer
        name: secret-operator-demonstration
      defaultCertificateLifetime: 1d
experimentalCertManager

Declares that the experimentalCertManager backend is used.

experimentalCertManager.issuer

The reference to the Cert-Manager issuer that should issue the certificates.

experimentalCertManager.issuer.kind

The kind of the Cert-Manager issuer, either Issuer or ClusterIssuer. Note that Issuer must be in the same namespace as the Pod requesting the secret.

experimentalCertManager.issuer.name

The name of the Issuer or ClusterIssuer to be used.

experimentalCertManager.defaultCertificateLifetime

The default duration of the certificates. This may need to be increased for backends that impose stricter rate limits, such as Let’s Encrypt.

kerberosKeytab

Format: Kerberos

Creates a Kerberos keytab file for a selected realm. The Kerberos KDC and administrator credentials must be provided by the administrator.

Only MIT Kerberos (krb5) and Active Directory are currently supported. Heimdal is not supported.

Principals will be created dynamically if they do not already exist.

The administrator keytab must have permission to add principals and get their keys. This corresponds to the flags ae in kadm5.acl.

Active Directory

Principal Conflicts

Each Active Directory domain should be used by only a single Kubernetes cluster.

This is because each pod, service, and node may be provisioned a principal matching its hostname, and principal names must be unique within a single AD domain. The Stackable Secret Operator will cache and reuse these credentials within a single Kubernetes cluster, but will not share them across multiple clusters.

If the same AD domain is shared between multiple Kubernetes clusters, the following must be unique across the AD domain:

  • The Kubernetes Nodes' names and fully qualified domain names

  • The Kubernetes Namespaces' names (only Namespaces that use Kerberos)

Custom samAccountName generation
samAccountName customization is an experimental preview feature, and may be changed or removed at any time.

By default, the samAccountName for created Active Directory accounts will be generated by Active Directory.

These can be customized if required, such as for compliance with internal policies. When customization is enabled, the name will follow the pattern {prefix}{random}, where {random} is a sequence of random alphanumeric characters. The full name will be exactly totalLength characters.

For example, the following configuration will generate samAccountNames such as "myprefix-abcd".

spec:
  backend:
    kerberosKeytab:
      admin:
        activeDirectory:
          experimentalGenerateSamAccountName:
            prefix: myprefix-
            totalLength: 13
kerberosKeytab.admin.activeDirectory.experimentalGenerateSamAccountName.prefix

A prefix that will be prepended to all generated samAccountName values.

kerberosKeytab.admin.activeDirectory.experimentalGenerateSamAccountName.totalLength

The desired length of samAccountName values, including prefix. Must not be larger than 20.

These options only affect newly created accounts. Existing accounts will keep their respective old samAccountName.

Reference

spec:
  backend:
    kerberosKeytab:
      realmName: CLUSTER.LOCAL
      kdc: krb5-kdc
      admin:
        mit:
          kadminServer: krb5-kdc
        # or...
        activeDirectory:
          # ldapServer must match the AD Domain Controller's FQDN or GSSAPI authn will fail
          # You may need to set AD as your fallback DNS resolver in your Kube DNS Corefile
          ldapServer: addc.example.com
          ldapTlsCaSecret:
            namespace: default
            name: secret-operator-ad-ca
          passwordCacheSecret:
            namespace: default
            name: secret-operator-ad-passwords
          userDistinguishedName: CN=Users,DC=sble,DC=test
          schemaDistinguishedName: CN=Schema,CN=Configuration,DC=sble,DC=test
      adminKeytabSecret:
        namespace: default
        name: secret-provisioner-keytab
      adminPrincipal: stackable-secret-operator
kerberosKeytab

Declares that the kerberosKeytab backend is used.

kerberosKeytab.realmName

The name of the Kerberos realm. This should be provided by the Kerberos administrator.

kerberosKeytab.kdc

The hostname of the Kerberos Key Distribution Center (KDC). This should be provided by the Kerberos administrator.

kerberosKeytab.admin.mit

Credentials should be provisioned in a MIT Kerberos Admin Server.

kerberosKeytab.admin.mit.kadminServer

The hostname of the Kerberos Admin Server. This should be provided by the Kerberos administrator.

kerberosKeytab.admin.activeDirectory

Credentials should be provisioned in a Microsoft Active Directory domain.

kerberosKeytab.admin.activeDirectory.ldapServer

An AD LDAP server, such as the AD Domain Controller. This must match the server’s FQDN, or GSSAPI authentication will fail.

kerberosKeytab.admin.activeDirectory.ldapTlsCaSecret

Reference (name and namespace) to a K8s Secret object containing the TLS CA (in ca.crt) that the LDAP server’s certificate should be authenticated against.

kerberosKeytab.admin.activeDirectory.passwordCacheSecret

Reference (name and namespace) to a K8s Secret object where workload passwords will be stored. This must not be accessible to end users.

kerberosKeytab.admin.activeDirectory.userDistinguishedName

The root Distinguished Name (DN) where service accounts should be provisioned, typically CN=Users,{domain_dn}.

kerberosKeytab.admin.activeDirectory.schemaDistinguishedName

The root Distinguished Name (DN) for AD-managed schemas, typically CN=Schema,CN=Configuration,{domain_dn}.

kerberosKeytab.adminKeytabSecret

Reference (name and namespace) to a K8s Secret object where a keytab with administrative privileges is stored in the key keytab.

kerberosKeytab.adminPrincipal

The name of the Kerberos principal to be used by the Secret Operator. This should be provided by the Kerberos administrator. The credentials for this principal must be stored in the keytab (adminKeytabSecret).

k8sSearch

Format: Free-form

This backend can be used to mount Secret across namespaces into pods. The Secret object is selected based on two things:

  1. The scopes specified on the Volume using the attribute secrets.stackable.tech/scope.

  2. The label secrets.stackable.tech/class specified in the Secret it’s self. This must match the name of the SecretClass.

Each field in this Secret is mapped to one file. It is suggested these Secret objects should follow one of the formats defined in this document.

In the example below, given the three object definitions for a Pod, a SecretClass and a Secret, the operator will first read the Pod’s volume attributes then look up the secret class. The k8sSearch backend will look up the Secret object labeled with secrets.stackable.tech/class: admin-credentials-class (the name of the secret class) and mount the fields of the Secret as files into the container at the specified mount point (/credentials). Please note that the contents in the volume will not update when the Secret content changes. A Pod restart is needed to refresh the Secret contents on disk.

apiVersion: apps/v1
kind: Pod
metadata:
  name: my-app
  labels:
    app.kubernetes.io/name: my-app
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: my-app
  containers:
    #... skipped for brevity
    volumeMounts:
    - name: admin-credentials-volume
      mountPath: /credentials
  volumes:
  - name: admin-credentials-volume
    ephemeral:
      volumeClaimTemplate:
        metadata:
          annotations:
            secrets.stackable.tech/class: admin-credentials-class
            secrets.stackable.tech/scope: pod
        spec:
          storageClassName: secrets.stackable.tech
          accessModes:
            - ReadWriteOnce
          resources:
            requests:
              storage: "1"
---
apiVersion: secrets.stackable.tech/v1alpha1
kind: SecretClass
metadata:
  name: admin-credentials-class
spec:
  backend:
   k8sSearch:
      searchNamespace:
        pod: {}
---
apiVersion: v1
kind: Secret
metadata:
  name: my-admin-credentials
  labels:
    secrets.stackable.tech/class: admin-credentials-class
stringData:
  user: admin
  password: secret

Scopes are translated into additional label filters of the form secrets.stackable.tech/$SCOPE: $SCOPE_VALUE. For example, a Pod named foo mounting a k8sSearch secret with the pod scope would add the label filter secrets.stackable.tech/pod: foo.

Reference

spec:
  backend:
    k8sSearch:
      searchNamespace:
        pod: {}
        # or...
        name: my-namespace
k8sSearch

Declares that the k8sSearch backend is used.

k8sSearch.searchNamespace

Configures the namespace searched for Secret objects.

k8sSearch.searchNamespace.pod

The Secret objects are located in the same namespace as the Pod object. Should be used for secrets that are provisioned by the application administrator.

k8sSearch.searchNamespace.name

The Secret objects are located in a single global namespace. Should be used for secrets that are provisioned by the cluster administrator.

Format

The k8sSearch backend doesn’t use a particular format, since the underlying Secret may contain any kind of free-form data. If a specific format is requested then the source format of the secret is inferred (based on the keys present in the secret), and a conversion is performed if required.

If a format is requested then only the keys relevant to the format will be mounted. For example, given a secret containing the files ca.crt, tls.crt, tls.key, and tls.config, the tls.config key will be projected to a file if no format is requested, but not if the TLS (PEM) format is requested.

Format

A format describes a set of artifacts (files and their respective contents) produced by a backend.

Each backend should conform to at least one common format. This is intended to allow cluster operators to switch between interoperable backends with minimal impact on secret consumers.

TLS (PEM)

Name: tls-pem

The secret contains the following files:

ca.crt

The certificate of the Certificate Authority (and associated chain) that has signed the certificate, in the PEM format.

tls.crt

The certificate identifying the Pod, in the PEM format.

tls.key

The private key corresponding to tls.crt, in the PEM format.

A TLS secret can also be converted into TLS (PKCS#12).

TLS (PKCS#12)

Name: tls-pkcs12

The secret contains the following files:

keystore.p12

A bundle of the private key and certificate (including the CA certificates), in the PKCS#12 format.

truststore.p12

A bundle of trusted CA certificates, in the PKCS#12 format.

Both stores are encrypted, with an empty string as the passphrase.

When using the k8sSearch backend, it is strongly recommended to store secrets in the use the TLS (PEM) format instead. The secret operator supports converting PEM key pairs into PKCS#12, but not the other way around.

Kerberos

Name: kerberos

The secret contains the following files:

krb5.conf

Kerberos configuration file for authenticating against the Kerberos realm.

keytab

A Kerberos keytab file containing credentials for all requested principals.