Enabling verification of image signatures

Image signing is a security measure that helps ensure the authenticity and integrity of container images. Starting with SDP 23.7, all our operator images are signed "keyless" (signing of product images and Helm charts will follow). By verifying these signatures, cluster administrators can ensure that the operator images pulled from Stackable’s container registry are authentic and have not been tampered with. Since Kubernetes does not have native support for verifying image signatures yet, we will use a tool called Kyverno in this tutorial.

Installing Kyverno

Kyverno can be easily installed via Helm:

helm repo add kyverno https://kyverno.github.io/kyverno/
helm repo update
helm install kyverno kyverno/kyverno -n kyverno --create-namespace

Other installation methods and options to run Kyverno in a highly-available fashion are described in the Kyverno documentation.

Creating a policy to verify image signatures

Now that Kyverno is installed, we can create a policy that verifies that all operator images that are part of the SDP 23.7 releases are signed by Stackable’s CI pipeline (Github Actions):

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: verify-image-signatures
  namespace: default
spec:
  validationFailureAction: Enforce
  webhookTimeoutSeconds: 30
  failurePolicy: Fail
  rules:
    - name: verify-stackable-signatures
      match:
        any:
          - resources:
              kinds:
                - Pod
      verifyImages:
        - imageReferences:
            - docker.stackable.tech/*-operator:23.7.*
          attestors:
            - entries:
                - keyless:
                    issuer: "https://token.actions.githubusercontent.com"
                    subject: "https://github.com/stackabletech/*-operator/.github/workflows/build.yml@refs/tags/23.7.*"
                    rekor:
                      url: https://rekor.sigstore.dev

Apply this policy to the cluster by saving it as kyverno-policy.yaml and running:

kubectl apply -f kyverno-policy.yaml

The policy will be applied to all namespaces in the cluster. It checks all newly created Pods that run any image matching the expression docker.stackable.tech/*-operator:23.7.* (all Stackable operators version 23.7.*) and ensures that these images have been signed by a Stackable Github Action from the release 23.7 (https://github.com/stackabletech/*-operator/.github/workflows/build.yml@refs/tags/23.7.*). If the signature of an operator image is invalid or missing, the policy will deny the pod creation. For a more detailed explanation of the policy options, please refer to the Kyverno documentation. If the subject field in the policy is changed to something like https://github.com/test/*, the policy will deny the creation of operator pods because the signature is no longer valid.