After four years of securing cloud native applications, our team at Aqua has learned a thing or two about applying best practices in the real world. We’ve seen many organizations succeed in establishing a sound process and tooling to achieve their security goals, and we’ve also seen those who struggle to prioritize and manage their efforts.
In this series of blog posts, we’ll review best practices for cloud native workloads security. We will focus on four core security concerns that Aqua CSP addresses. This first blog covers vulnerability management best practices.
Vulnerability Management for Containers and Serverless
Gaining visibility into vulnerabilities in code is key to reducing the attack surface and eliminating risk. It’s a place where many organizations start their cloud native security journey, since it’s a manageable “win” and is visible in the development stage, making it suitable as a first step to secure applications before they get to production.
In container images and functions that are developed using CI/CD methods, there are some unique challenges that must be addressed for effective vulnerability management:
- Discovering vulnerable images and functions: The sooner you can find and fix vulnerabilities, the better. This means scanning images and functions in the CI pipeline, as well as in registries or function stores. Another area that’s often overlooked are container images stored directly on hosts (VMs) that are not placed in registries.
- Prioritizing vulnerabilities by overall risk: For example, a high severity vulnerability that affects a component that isn’t being used might be less risky than a medium severity vulnerability that’s instantiated in 100 containers in production.
- Eliminating false positives: Unfortunately, relying purely on a public database like NVD (the National Vulnerability Database) isn’t very accurate. NVD is updated often with a 4-5-week delay and includes many false positives – CVEs that appear more severe than they really are or have been fixed elsewhere. The right context can help eliminate false positives and requires reconciling multiple sources and some research.
- Understanding the vulnerable components and their owners: Image structure isn’t flat and may include dependencies, base images and multiple Pinpointing the root source of a vulnerability is key to getting it fixed and to avoiding a single vulnerability being flagged as different ones.
- Getting remediation information: You found a vulnerability in an image – how do you fix it? Is there a fix? What should a developer do?
- Handling exceptions and protecting against unfixable vulnerabilities: In a perfect world, we shouldn’t run with vulnerabilities. Reality is different, and many organizations knowing run applications that are vulnerable, simply because they have no other choice. The question is how one handles these exceptions and mitigates vulnerabilities that can’t be fixed within a reasonable timeframe.
- Automating the entire process and scaling it: With many organizations now scanning 10s of thousands of images per day, it’s not at all guaranteed that a tool that works for a few images will be able to handle such massive numbers.
There are several tools on the market (including open source tools) that scan images for vulnerabilities – however, they vary greatly in their coverage, accuracy, and of course their ability to address the outcomes. As a customer once told me, it’s great to know we have so many vulnerabilities, now I’m losing sleep. But what do I now do with all that information?
Let’s look at some best practices and how an enterprise-grade platform like Aqua CSP applies them out of the box.
Scanning Image Registries and Functions
It’s recommended to start with scanning as early in the SDLC as you can. Aqua integrates with virtually all CI/CD tools to facilitate scanning as images and functions are being built, detect issues early and allow for quick remediation.
Then, scanning registries should be done continuously, since even images or functions that were OK’d as they were being built, may have newly discovered vulnerabilities in them, or have images pushed into a registry that bypassed the CI process. Many organizations use multiple CI tools and multiple registries (for different teams or stages), and all should be scanned. Aqua CSP provides out of the box connectivity with all registries from Docker Trusted Registry, through Amazon ECR, Azure ACR, Google GCR, Harbor, JFrog Artifactory, Sonatype Nexus, Red Hat Quay, etc.
Focus on Application Risk
The next challenge you’ll face is providing security at scale. Scale, in this case, refers to potentially hundreds of thousands of images scanned. The sheer volume of images results in an overwhelming number of vulnerabilities that need to be prioritized for remediation. Registries may maintain images that are in development or deployed to staging or production environments throughout the application lifecycle. But many of these images are not actually deployed in production, and so pose a lower risk even if vulnerable. Aqua CSP provides visibility into instantiation of vulnerable components in running workloads, which allows security teams to focus on fixing those components that are most exposed to risk of exploitation.
CVEs in running images
Reduce False Positives with Image Vendor Scoring
The Aqua CSP scoring system for vulnerabilities is based on the CVSS vulnerability score, as reported by the National Vulnerabilities Database (NVD), combined with the package vendor. To minimize false positives in cases where there are conflicts between the NVD reports of a CVE’s severity and the vendor’s report, the vendor score takes precedence. In many cases the vendor score will be lower as the vendor reduced the risk of the CVE through means other than fixing the vulnerable code. In other cases, as in the example below, the vendor rated the CVS as more severe than NVD rated it:
Identify the Application Owner
Ownership visibility is critical for an effective and efficient vulnerability remediation plan. To help identify the application’s owner, Aqua CSP displays the image author in the image maintainer tag, often with an e-mail address.
Go to the Base Image
Many images are based on other images, which means they inherit their vulnerabilities. It is important to understand the origin of these vulnerabilities, because they can appear in multiple images, all streaming from the same base image. Fixing the base image would therefore fix those vulnerabilities everywhere they are used. Aqua CSP automatically displays base image vulnerabilities where they exist, isolating their source and enabling a more efficient remediation.
Viewing Vulnerabilities by Layer
It’s often the case that an image was perfectly clean until one update adds a layer that contains vulnerable packages. By using the Aqua CSP layer view, you can pinpoint the layer that is associated with the vulnerability you want to address. In addition, if the vulnerable package has an available fix, Aqua CSP will include that as a suggested solution.
Aqua CSP Layer View
Getting Actionable Advice
When developers are informed about vulnerabilities in their code (and they will often get a bulk of those tickets opened), they are often stumped about the next step. In Aqua CSP we provide, as part of the vulnerability information, actionable and specific advice on what should be done to remediate the issue. Usually, an upgrade to a specific newer version will be advised, or possibly changes in configuration or environment variables.
CVEs and Suggested Solutions Help You Fix Vulnerabilities
Vulnerability Identified: Should I Remediate, Mitigate, or Accept?
In an ideal world, all vulnerabilities should be remediated. In an immutable cloud native universe this means going back to the source image or function, and fixing the vulnerable component (usually by updating it to a fixed version). Unfortunately, it isn’t always possible to do that – e.g., when a fix is simply not available, or when there are many dependencies that would require the fix to take a long time.
What do you do in the meantime? Aqua has come up with two options for this – first, you can use Aqua Vulnerability Shields to automatically apply runtime controls that prevent the exploitation of a vulnerability. In case this is not possible or applicable, you have the option of temporarily accepting the risk of a vulnerability by choosing Acknowledge and setting an expiry date on it. This will give developers the grace period they need to seek remediation options.
Aqua CSP vShield Policy Page
Integrate Aqua with Your Centralized Vulnerability Management System
You can use Aqua CSP to generate comprehensive reports that illustrate vulnerability risks. These reports answer questions such as: What should be remediated first? Is there an available fix? and “Who is the owner of the application? Most enterprises pursue for a single pane of glass for centralized security management, assessment, and prioritization. To achieve that, you can use our API integration or CSV reports to export Aqua CSP reports to third-party systems.
Aqua CSP Splunk integration configuration screen
Up Next
Stay tuned for our next posts in the series Cloud Native Security Best Practices. We’ll cover one of the biggest challenges in Cloud Native environments and discuss how to gain visibility and assess the security of running workloads. You will also learn how to secure the CI/CD pipeline, and how to set a security gates for your pipeline without affecting productivity.