DevSecOps

DevSecOps with Trivy and GitHub Actions

The premise of DevSecOps is that in the Software Development Life Cycle (SDLC), each member is responsible for security. This unifies the operations and development teams in terms of security operations. DevSecOps’ goal is to add security to each step of the development process by integrating security controls and processes as early as possible in the DevOps process.

In this post, I’ll describe how to configure a useful DevSecOps workflow for a simple microservice implemented in Golang. The source code repository of the application leverages GitHub Actions to build a Docker container and scan it for vulnerabilities with Trivy on each push to the master, or a feature branch.

I will also demonstrate how to release application binaries built into a container and publish such a container to a GitHub Package Registry.

GitHub Action and Workflows

GitHub Actions allow the creation of custom software development life cycle workflows directly in a GitHub repository. Workflows are automated processes that you can set up to build, test, package, and release any code project on GitHub. In other words, you can build end-to-end continuous integration (CI) capabilities directly in your repository. Similarly, you can build continuous deployment (CD) pipelines, but we’ll focus just on CI in this post.

We’ll create two workflows:

1. Build workflow is for triggering snapshot builds on push to master or pull request branch. This will checkout code, run unit tests, upload code coverage reports, and build a release snapshot. Each release snapshot is an application binary, plus the Docker image containing that binary. The Docker image is tagged with the Git commit hash and then scanned with Trivy for vulnerabilities. The scan fails the build if there are any critical vulnerabilities found.

2. Release Workflow is for release builds. Whenever you create an annotated tag on a master branch, the Release workflow is triggered to run unit tests and build the release binary and the Docker image. This time, the Docker image is tagged with the release version. If everything is okay, the build artifacts are uploaded to the GitHub repository’s releases page and to GitHub Package Registry. After a successful push to the registry, the image is scanned by Trivy again to make sure that the released containers are not vulnerable.

As you’ll see, whether we build a snapshot or a release it’s very easy to configure Trivy and detect vulnerabilities early on in the development stage. This gives you more time to fix security issues by upgrading or replacing vulnerable dependencies.

In the subsequent sections, I’ll walk you through implementation details of Build and Release workflows configured for the sample application.

GitHub Project Settings

Before going into the implementation details, let’s review the GitHub project’s configuration. Note that GitHub Actions are not triggered on forked repositories. In order to run the examples described in this post, you must set up your own project based on the dev-sec-ops-seed repository. Make sure to add the following secrets to the project's settings:

  • GITHUB_REGISTRY_USER – Your GitHub identifier
  • GITHUB_REGISTRY_TOKEN – Your personal GitHub access token with scope for publishing releases and packages

At the time of writing, GitHub Package Registry is available in public beta, which means that you must request access to use this service.

GoReleaser

GoReleaser is a release automation tool for Go projects. It simplifies the build, release and publish steps while providing customization options for all steps. It’s built for CI tools so you can download and execute it in your build script. There’s even an official GoReleaser Action for GitHub Actions.

You can customize your release process through the goreleaser.yml file. In the sample project, GoReleaser is configured to compile Go source code and build Docker images.

GoReleaser can be run in test mode which skips artifact publishing. This feature is leveraged in the snapshot build to  make sure that GoReleaser is configured properly.

$ release --snapshot --skip-publish

It’s also run for release builds where it does the actual release and publishing to GitHub.

$ release

At this point, you can see that GoReleaser lacks an ability to configure a pre-push hook for Docker builds which would allow us to scan the image before pushing it to the GitHub Package Registry. I’ve even requested such a feature and you can track it here; Add support for Docker pre-push hook.

In the sample project, the build workflow is configured to scan locally cached Docker images, whereas the release workflow triggers scanning only after the image is pushed to the GitHub Package Registry. This is not ideal, especially when some registries, such as GitHub Registry, do not support simple artifact removal.

The Build Workflow

The Build workflow has a few steps. I’ll only focus on the steps relevant to security scanning, as the rest is described in GitHub’s documentation.

The first interesting step is called Release snapshot. It validates GoReleaser’s configuration and builds the application’s binaries and the Docker image containing that binary. GoReleaser is configured to tag the locally cached Docker image with a hash of the Git commit triggering the build.  Note that because of the  --snapshot and --skip-publishing flags passed to GoReleaser, it does not publish any build artifacts to GitHub yet.

The actual scanning for vulnerabilities is done in the Scan image for vulnerabilities step by configuring a GitHub Docker container action, which refers to the official Docker image of Trivy, i.e., docker.io/aquasec/trivy:0.2.1.

Here is a sample log output of a snapshot build which failed because of medium and low vulnerabilities found in the alpine:3.10.2 image, specifically, the openssl package.The Docker command and Trivy’s flags are passed as the action’s arguments. The --exit-code 1 flag marks the build as failed if there are any vulnerabilities found by the scanner. In addition, we could use the --severity CRITICAL flag to only fail builds when critical vulnerabilities are found. Note that the expression allows us to refer to the locally cached Docker image build in the Release snapshot step.

2-trivy_OK

There’s one more noteworthy flag that we passed as an argument to the Trivy action, which is cache-dir. It’s the POSIX file system location where Trivy clones its vulnerabilities database. The database itself is a Git repository, and the initial clone takes a significant amount of time, but subsequent scans are faster. Unfortunately, at the time of writing this post, GitHub Actions don’t support directory caching between workflow executions. However, such a feature is expected to be available by mid-November, 2019. If it is available by the time you’re reading this post, you can mark the /var/lib/trivy directory as cached directory and enjoy faster scanning by cache.

The Release Workflow

Once we make sure that the master branch is stable and a snapshot Docker image is built without any vulnerabilities detected, we can release a new version of the application by creating an annotated tag in Git.

If you look at the Release workflow definition, it’s even more simple than the Build workflow. It configures nearly the same GoReleaser Action to build the binaries and the Docker image, but this time it does the actual publishing to GitHub. Also, the Docker image is tagged with the release version rather than a Git commit hash. For example, docker.pkg.github.com/<your github id>/dev-sec-ops-seed/seed:<release version>.

Notice that the Release step is preceded with the docker login command to authorize the workflow runner with the GitHub Package Registry. Otherwise, the docker push command executed by GoReleaser will fail.

As an example, to release a new version of the application run:

$ git tag -a v0.0.6 -m "Release v0.0.6"
$ git push origin v0.0.6

This will trigger the Release workflow. If everything goes well, you can pull the image with docker pull docker.pkg.github.com/<github id>/dev-sec-ops-seed/seed:0.0.6.
You can also download compiled Golang binaries from https://github.com/<github id>/dev-sec-ops-seed/releases.

Summary

DevSecOps is the philosophy of developing applications securely from ideation to deployment. Modern applications are typically split into multiple microservices which are put into containers. Those containers include runtime dependencies of a given microservice, like shared libraries, helper binaries, and more. Docker is one of the popular choices for building, publishing and running containers, whereas Kubernetes is becoming the de facto standard for deploying, scaling, and managing containerized application. Before even publishing a container to a Docker registry accessible to Kubernetes, it’s crucial to make sure that the corresponding microservice and its dependencies are not vulnerable.

Make sure that your CI/CD pipeline for building containerized applications check for vulnerabilities. There are no excuses, when tools such as Trivy or GitHub Actions are out there!

Trivy

Picture of Daniel Pacak

Daniel Pacak

Daniel Pacak is an Open Source Engineer at Aqua Security. He works on Kubernetes and container security related projects, while also taking part in maintaining the CNCF's project, Harbor. When he isn't at work, he enjoys taking walks in the woods with his family, dog, and horse.

Container Security, DevSecOps