A Zero Trust Approach for Securing the Supply Chain of Microservices Packaged as Container Images

Securing the software supply chain has become a top priority for both open source as well as closed source projects. There are many sources in a software supply chain such operating systems and operating system packages/binaries, programming language frameworks and libraries, shell scripts, source code, container images and many others. Since container images are becoming so prevalent with the popularity of the Kubernetes platform, we will address securing container images in this blog.

A common pattern I have come across in projects that containerize microservices is one in which software development teams would create a “golden” base container image and any microservice created would be built using that golden base image as depicted in the diagram below.

base image as a basis for microservices

One of the main reasons development teams adopt this approach is to speed up the build process for building microservice container images. For example:

• Base image has key components such as OS, language frameworks, key libraries, and common code
• Base container image is stored in a secure private container registry close to the rest of the infrastructure therby lowering latency

This blog will propose a solution on how to secure the supply chain for the pattern described above, specifically focusing on securing the container image supply chain.

In our scenario, we want to build a Quarkus java microservice called the quarkus-hello-service that runs on the OpenShift Container Platform. We first create a quarkus base image and then we build and run the quarkus-hello-service microservice.

quarkus-hello-service container image built from quarkus-base-image

The key supply chain security requirements that we will address are:

• quarkus-base-image container image should be digitally signed so that it can be verified by consumers
• quarkus-hello-service container image is built from the secure and verified quarkus-base-image
• qaurkus-hello-service container image should be digitally signed so that it can be verified by consumers
• quarkus-hello-service microservice should only run on the Kubernetes/OpenShift platform if its image has a valid digital signature.
• Attestation or evidence of the build process

Description of the Solution

The following diagram shows the key components of the overall solution. The solution leverages new open source tools that have been developed by the Sigstore project for digitally signing and verifying container images. The solution also leverages the Kyverno policy engine which is a Kubernetes admission controller that has integration with the Sigstore tools to validate container images before allowing them to run on a cluster.

Zero trust supply chain security solution for container images

The quarkus-base-image Tekton pipeline fetches the source code from github containing the Dockerfile that has the instructions to build the base image. The instructions specify to use the minimul Red Hat Universal Base Image version 8.3 (ubi8/ubi-minimal:8.3) as well as the Java 11 runtime JDK.

FROM registry.access.redhat.com/ubi8/ubi-minimal:8.3
ARG JAVA_PACKAGE=java-11-openjdk-headless
ARG RUN_JAVA_VERSION=1.3.8
ENV LANG='en_US.UTF-8' LANGUAGE='en_US:en'
# Install java and the run-java script
# Also set up permissions for user `1001`
RUN microdnf install curl ca-certificates ${JAVA_PACKAGE} \
&& microdnf update \
&& microdnf clean all \
&& mkdir /deployments \
&& chown 1001 /deployments \
&& chmod "g+rwX" /deployments \
&& chown 1001:root /deployments \
&& curl https://repo1.maven.org/maven2/io/fabric8/run-java-sh/${RUN_JAVA_VERSION}/run-java-sh-${RUN_JAVA_VERSION}-sh.sh -o /deployments/run-java.sh \
&& chown 1001 /deployments/run-java.sh \
&& chmod 540 /deployments/run-java.sh \
&& echo "securerandom.source=file:/dev/urandom" >> /etc/alternatives/jre/conf/security/java.security

The pipeline invokes Tekton tasks that build the docker image and push the image (docker.io/gkovan/quarkus-base-image:1.0) to the specified container repository. Tekton Chains is installed in the cluster. Tekton Chains observes the Tekton TaskRuns and digitally signs any derived images as well as the TaskRun itself. By signing the TaskRun, undeniable evidence exists on the build details itself such as input parameters, output parameters, and the builld logic. Tekton Chains has integration with Sigstore cosign tool to sign the container images as well as all the TaskRuns executed. At this point, the signed quarkus-base-image exists in our container registry ready to be used. This base image has a secure mimimal operating system (linux) as well as a JDK 11 runtime.

The hello service implements a single REST endpoint called “/hello”. The quarkus-hello-service Tekton pipeline fetches the source code from github which contains the java Quarkus code as well as the Dockerfile with instructions on how to build the quarkus-hello-service image. The pipeline first builds the java code using maven in the “build source code” task. The pipeline then proceeds to build the image using the Dockerfile in the repo. The Dockerfile references the quarkus-base-image and then copies the Quarkus libraries and application code into the image.

FROM docker.io/gkovan/quarkus-base-image:1.0
LABEL base.image="docker.io/gkovan/quarkus-base-image:1.0"
ENV JAVA_OPTIONS="-Dquarkus.http.host=0.0.0.0 -Djava.util.logging.manager=org.jboss.logmanager.LogManager"
COPY --chown=1001 target/quarkus-app/lib/ /deployments/lib/
COPY --chown=1001 target/quarkus-app/*.jar /deployments/
COPY --chown=1001 target/quarkus-app/app/ /deployments/app/
COPY --chown=1001 target/quarkus-app/quarkus/ /deployments/quarkus/
EXPOSE 8080
USER 1001
ENTRYPOINT [ "/deployments/run-java.sh" ]

The “build/push docker image” task then pushes the resulting image (docker.io/gkovan/quarkus-hello-service:1.0) to the image repository. Tekton chains, digitally signs the image as well as all the TaskRuns executed in the pipeline. The “verify base image” task, validates that the base image “docker.io/gkovan/quarkus-base-image:1.0” is signed. The task uses the cosign tool that is part of the Sigstore project to do this validation. The cosign tool has access to the public key (configured as a kubernetes secret) corresponding to the private key that was used to sign the image, to do the validation. Since the base image is part of the supply chain, it is critical that we validate the signature to ensure that it has not been tampered with. The “deploy” task in the pipeline deploys the microservice to the OpenShift cluster by applying the appropriate Kubernetes manifest files. All requests to apply kubernetes manifest files go through the kube-apiserver component. The kube-apiserver first checks with all the admission controllers installed in the cluster to see if the request is valid. The Kyverno admission controller has a check-image policy that has been configured to only allow valid images. The policy includes a pattern matching rule for images as well as the public key to validate the images as follows:

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
name: check-image
spec:
  validationFailureAction: enforce
  background: false
  rules:
    - name: check-image
      match:
        resources:
          kinds:
            - Pod
      verifyImages:
      - image: "docker.io/gkovan/*"
        key: |-
        -----BEGIN PUBLIC KEY-----
    MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEtwDUUcAVE4dJj623tCNy9WrYfgJn
gPqTAl4hkXdXMG1jk36OGxzwefjQO7XV37i0kJrWssfggVGxa0GoXryWVw==
        -----END PUBLIC KEY-----

If the quarkus-hello-service image is not signed or cannot be validated with the specified public key, then the Kyverno admission controller will tell the kube-apiserver that the request is not valid and the deployment will fail. This ensures that only trusted container images are allowed to run on the cluster providing the runtime security.

Summary

This blog discussed a solution for securing the supply chain of a typical microservices project focusing on zero trust for container images. The solution discussed how the container images that are built are signed and verified during the build process. Furthermore, the solution discussed how to ensure only valid and signed container images are allowed to run on the OpenShift/Kubernetes cluster. Another important aspect is to be able to provide attestation and evidence of the build process such as the inputs, outputs and logic to produce and run the software artifacts. The overall solution leveraged tools from the Sigstore project for digitally signing the software container images as well as the Kyverno admission controller for only allowing valid signed images to run on the OpenShift cluster.

All source code can be found in the following github repos:

quarkus-base-image: https://github.com/ztsc/quarkus-base-image

quarkus-hello-service: https://github.com/ztsc/quarkus-hello-service

OpenShift Pipeline/Tekton resources: https://github.com/ztsc/tekton

Kyverno check-image policy: https://github.com/ztsc/admission-controller/blob/master/kyverno/check-image-cluster-policy.yaml

Key References:

Project Sigstore, https://www.sigstore.dev/

Sigstore cosign tool, https://github.com/sigstore/cosign

OpenShift Pipelines/Tekton, https://docs.openshift.com/container-platform/4.7/cicd/pipelines/understanding-openshift-pipelines.html

Tekton Chains, https://github.com/tektoncd/chains

Kyverno, https://kyverno.io/

OpenShift Container Platform, https://docs.openshift.com/container-platform/4.7/welcome/index.html

Quarkus, https://quarkus.io/