Enhancing Your GitHub Workflow with Docker Scout

In the past few days, I've been watching a few talks about best practices in the Secure Software Supply Chain. I particularly enjoyed this one from Kelsey Hightower. These best practices cover a very wide range of the development lifecycle, shifting security to the left:

1. Sign your Git commits. Anyone can push commits to their own repos with anyone else's email address, that's why signing your Git commits protects you against someone impersonating you.
2. Scan your container image for vulnerabilities.
3. Generate SBOM (Software Bill Of Materials) and provenance attestations. The former is useful to identify all the components, libraries, and dependencies that make up your software - think of it as a list of ingredients - whereas the latter describes how the software was built.
4. Container image signing ensures the authenticity and integrity of the image. The signature is used to verify the image was not tampered.
5. Manage policies to allow or block specific container images to run in your Kubernetes cluster.

Incorporating and automating some of these Software Supply Chain security best practices in your CI pipeline can be made easier. Check out below my opinionated GitHub workflow to get started.

A quick poll

I was genuinely curious about how many of you accomplish some of the best practices aforementioned, so I decided to run a poll on Twitter for a couple of days.

First of all, thank you to the 175 people who participated in the poll and who shared it with their network. Obviously, these results are not representative enough to speak for the whole developer community who make efforts to secure their Software Supply Chain, but at least it gives me a rough overview of what efforts developers in my network are accomplishing in their CI pipelines.

What do we do next?

On one hand, if you happen to be one of those individuals who don't know where to start in securing your software supply chain, in this post I want you to learn how to get started without investing a lot of time and effort. On the other hand, if you already do a combination of some of them, you may still learn something new 😉.

I'm going to introduce you to an opinionated GitHub workflow that I use for my personal projects to help you build and release more secure software. As a high-level overview, it focuses on covering the following best practices:

• Generate SBOM and provenance attestations with BuildKit.
• Run a vulnerability analysis of the container image with Docker Scout.

Notice I use the term vulnerability analysis instead of scanning on purpose when referring to Docker Scout, as it follows a novel event-driven model which is different from traditional vulnerability scanners. Whenever a new vulnerability affecting your images is announced, Scout shows your updated risk within seconds.

The source code of my GitHub workflow template is here. You can create a repo from it or look at the workflow itself to adapt your existing ones.

A high-level overview of the GitHub Workflow

Overview

The workflow consists in building a container image with SBOM and provenance attestations that don't contain any critical or high vulnerabilities. Consequently, it expects a Dockerfile to be present in the root of your Git repository. Depending on how the workflow is triggered, we can differentiate the following two scenarios:

Whenever a PR is opened, or subsequent commits are pushed to it, Docker uses the Dockerfile to build the container image with the new changes for a single platform (i.e. linux/amd64). Then, the image is analyzed for critical and high vulnerabilities that have a fix available and compared side-by-side against the edge tag. (The edge tag reflects the last commit of the active branch on your Git repository, usually main).

Whenever a PR is merged, the container image is built for a single platform – linux/amd64 – and analyzed for vulnerabilities. If no critical or high vulnerabilities are found, the image is built for multiple platforms – linux/amd64 and linux/arm64 – and tagged under the names edge and the short sha (e.g. 76adad1). Ultimately, these two tags are pushed to DockerHub alongside their SBOM and provenance attestations.

Analyze the image for vulnerabilities

I use the docker/scout-action to carry out the vulnerability analysis to find out whether the recently built image contains vulnerabilities and report the results in a comment in the PR.

      - name: Docker Scout CVEs
        uses: docker/scout-action@v0.18.1
        with:
          command: cves
          image: "" # If image is not set (or empty) the most recently built image, if any, will be used instead.
          only-fixed: true
          only-severities: critical,high
          write-comment: true
          github-token: ${{ secrets.GITHUB_TOKEN }} # to be able to write the comment
          exit-code: true

The docker/scout-action is very flexible and allows you to customize some aspects of the image analysis through the parameters of the GitHub action. In the example above, I'm interested in displaying critical and high vulnerabilities with a fix available (only-fixed: true). Also, in the case of detecting vulnerabilities, I want the GitHub workflow to fail (exit-code: true).

For instance, the image below depicts the report of the vulnerabilities and some other metadata information found by Docker Scout as part of a regular workflow run in the PR.

Some relevant image metadata that is reported includes the digest, platform, size, and number packages. Also, there's a breakdown of vulnerabilities found in the base image of my Dockerfile with remediation advice, i.e. the version of the package that contains the fix.

Compare the image against the edge tag

As days go by, new CVEs will be discovered and the edge tag in the registry may contain some of these new CVEs.

Using the docker/scout-action with the command compare will tell us whether the new changes introduced via a PR result in fixing or adding more new vulnerabilities compared to the edge tag which is already available in the registry.

- name: Check if ":edge" tag exists
  if: github.event_name == 'pull_request'
  id: check
  continue-on-error: true
  run: |
    docker buildx imagetools inspect ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:edge

- name: Docker Scout Compare image against ":edge" tag
  if: always() && steps.check.outcome == 'success' && github.event_name == 'pull_request'
  uses: docker/scout-action@v0.18.1
  with:
    command: compare
    image: ${{ steps.meta.outputs.tags }}
    to: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:edge
    only-fixed: true # Filter to fixable CVEs only
    ignore-unchanged: true # Filter out unchanged packages
    only-severities: critical,high
    write-comment: true
    github-token: ${{ secrets.GITHUB_TOKEN }} # to be able to write the comment
    exit-code: true # Fail the build if vulnerability changes are detected

For instance, in the example below I opened a PR to fix the vulnerabilities present in the edge tag. It contains 4H 10M vulnerabilities that are fixed by upgrading the base image from alpine:3.17 to alpine:3.18.2.

Once the PR is merged, the GitHub workflow will be triggered for the main branch, building and pushing a new image that effectively does not contain any vulnerabilities. We can confirm it by looking at the image summary in DockerHub:

Generation of SBOM and provenance attestations

The docker/build-push-action allows you to generate SBOM and provenance attestations just by using sbom: true and provenance: true.

- name: Multi-platform Docker Build and Push to registry
  id: build-and-push
  uses: docker/build-push-action@v4
  with:
    push: ${{ github.event_name != 'pull_request' }}
    sbom: true
    provenance: true
    tags: ${{ steps.docker_meta.outputs.tags }}
    labels: ${{ steps.docker_meta.outputs.labels }}
    cache-from: type=gha
    cache-to: type=gha,mode=max
    platforms: linux/amd64,linux/arm64

These attestations are generated at the end of the build process and attached to the image manifest index when the image is pushed to the registry.

Using tools such as explore.ggcr.dev, docker buildx imagetools inspect, or crane allows you to visualize and navigate through the image manifest layers:

To learn more about SBOM and provenance attestations in BuildKit check out my other blog post.

Conclusion

There are multiple ways in which you can make your Software Supply Chain more secure. Git commit signing, SBOMs and provenance attestations, vulnerability analysis, and container image signing seem to be the most relevant practices nowadays.

In this post, I have provided an opinionated GitHub workflow that you can use straight away and which takes into account several of the best practices mentioned.

Finally, it's evident that Docker is improving the UX of existing tools such as buildx generating the SBOM and provenance attestations as part of the docker build command. Also, recent emerging tools such as docker scout play an important role in providing feedback about how secure your container image is in the very early process of the developer's inner loop.

Disclaimer: The content I have contributed to this blog post is my own and does not necessarily represent the views or opinions of my employer.