Enabling verification of image signatures

Image signing is a security measure that helps ensure the authenticity and integrity of container images. Starting with SDP 23.7, all our images are signed "keyless". By verifying these signatures, cluster administrators can ensure that the images pulled from Stackable’s container registry are authentic and have not been tampered with. Since Kubernetes does not have native support for verifying image signatures yet, we will use a tool called Kyverno in this tutorial.

Releases prior to SDP 23.7 do not have signed images. If you are using an older release and enforce image signature verification, Pods with Stackable images will be prevented from starting.

Installing Kyverno

Kyverno can be easily installed via Helm:

helm repo add kyverno https://kyverno.github.io/kyverno/
helm repo update
helm install kyverno kyverno/kyverno -n kyverno --create-namespace

Other installation methods and options to run Kyverno in a highly-available fashion are described in the Kyverno documentation.

Creating a policy to verify image signatures

Now that Kyverno is installed, we can create a policy that verifies that all images provided by Stackable are signed by Stackable’s CI pipeline (Github Actions):

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: verify-image-signatures
  namespace: default
spec:
  validationFailureAction: Enforce
  webhookTimeoutSeconds: 30
  failurePolicy: Fail
  rules:
    - name: verify-stackable-signatures
      match:
        any:
          - resources:
              kinds:
                - Pod
      verifyImages:
        - imageReferences:
            - docker.stackable.tech/*
          attestors:
            - entries:
                - keyless:
                    issuer: "https://token.actions.githubusercontent.com"
                    subject: "https://github.com/stackabletech/*/.github/workflows/build.yml@refs/*"
                    rekor:
                      url: https://rekor.sigstore.dev

Apply this policy to the cluster by saving it as kyverno-policy.yaml and running:

kubectl apply -f kyverno-policy.yaml

The policy will be applied to all namespaces in the cluster. It checks all newly created Pods that run any image matching the expression docker.stackable.tech/* (all images provided by Stackable) and ensures that these images have been signed by a Stackable Github Action (https://github.com/stackabletech/*/.github/workflows/build.yml@refs/*). If the signature of an image is invalid or missing, the policy will deny the pod creation. For a more detailed explanation of the policy options, please refer to the Kyverno documentation. If the subject field in the policy is changed to something like https://github.com/test/*, the policy will deny the creation of pods with Stackable images because the signature is no longer valid.