What if you could prevent an attack like SolarWinds in a few easy steps? Since sophisticated software supply chain attacks usually hide in legitimate build processes and code updates, they’re often missed by regular code scanning tools. To counter these threats, we’re excited to announce pipeline integrity scanning — an innovative eBPF-based capability designed to provide integrity assurance for your build pipeline. Aqua customers can now detect and block suspicious behavior and malware in their dev environments, thereby enhancing protection against advanced supply chain attacks.
Realizing the critical need for build integrity
Today’s complex software supply chains are prime targets for hackers as they provide an opportunity to reach a large number of potential victims through one single breach. In the past few years, attacks such as SolarWinds and Codecov showed how terrifying a compromise to the integrity of the software build process can be.
The dramatic magnitude of these attacks has changed the paradigm on how to secure applications. Because adversaries aim to compromise build and development pipelines by inserting malicious processes in the legitimate software itself, traditional code scanning is no longer enough and is not able to detect such threats.
This has led to significant investments in specialized solutions and security improvements across industry and governments. Software supply chain security has become a high priority focus for every organization. Yet, while there has been much debate along with updates to systems, processes, and regulations, supply chain attacks continue to rise.
Gartner predicts that by 2025, 45% of organizations worldwide will have experienced attacks on their software supply chains, a three-fold increase from 2021.
On top of that, the critical need to continuously validate the software integrity is also driven by industry compliance requirements. Organizations are being held accountable for incorporating security best practices throughout their software development lifecycles. Integrity is mentioned as one of the key requirements in major industry frameworks for software supply chain security, such as SLSA, NIST SSDF, and CIS Software Supply Chain Benchmark.
Along with recommending traditional scanning for vulnerabilities and misconfigurations, these guidelines emphasize the importance of assuring software integrity, which is more difficult for organizations to achieve. In this sense, integrity means validating that your build process operates as expected and your code hasn’t been tampered with.
The bottom line is — yes, you need to ‘shift left’ to fix vulnerabilities in your code as early in the application lifecycle as possible. But to produce inherently secure software, it’s now more important than ever to also protect the development process itself and ensure that your software hasn’t been modified by malicious actors.
Enter pipeline integrity scanning
Preventing sophisticated supply chain attacks as mentioned above is a complex challenge. To help tackle it, we built pipeline integrity scanning. It works by scanning your CI/CD pipeline with eBPF-based technology and detecting in real time suspicious behavior or malware that might indicate a supply chain attack.
This innovative capability relies on several core pieces:
- The build process has its “own version” of runtime — in order for the build step to happen, the pipeline leverages runners and triggers dynamic activity. So, the build can be secured just as any other runtime environment. Being a recognized runtime security player in the cloud native ecosystem, Aqua is best positioned to bring this knowledge and technology to build pipelines as well.
- Power of eBPF — under the hood, integrity scanning leverages Tracee, Aqua’s robust open source runtime security and forensics tool for Linux. Thanks to its lightweight capabilities, eBPF technology can provide visibility into the build’s runtime and detect threats in real time with minimal disruption.
- Behavioral analysis – the capability also takes advantage of behavioral signatures produced by the Aqua Nautilus research team to detect zero-day threats based on cloud native attacks seen in the wild. Because of this, we can detect multiple malicious techniques, including fileless execution, files downloaded from a remote resource and executed straight from memory, reverse shell execution and executables dropped to memory.
In general, the idea behind integrity scanning is simple. First, we determine the baseline of how your build process operates and then we treat anything outside of it as suspicious. A good analogy would be observing human non-verbal communication. For example, it’s commonly believed that if you blink too often in a conversation, you might be lying or holding something back. But that’s not the case: behavior can be considered suspicious only when it deviates from that particular person’s baseline.
How integrity scanning works
So how does this work in practice? Let’s take a look.
Figure 1 – Deploying a pipeline enforcer on the GitHub build
Connect to the build pipeline. First, you need to add the ‘pipeline enforcer’ to your build pipeline, for example, on GitHub.
- Monitor the build pipeline. Once the integration is complete, we need to understand how your build pipeline is run and what is typical and normal for your specific environment. So we start monitoring key activities within the pipeline — what files are created and modified, where it typically communicates to, what network calls are invoked, which containers and processes run, etc.
- Define the baseline. Modern build pipelines are very dynamic and complicated, involving a lot of processes, files, communication, calls and more. However, by monitoring it for a certain period of time, we are able to understand the baseline of your pipeline, even with all its dynamics and speed.
This baseline represents a known, good and secure state in the context of your specific environment. - Detect any drifts from the baseline. Once the baseline is established, we can detect any drifts from this state and alert you on anything unusual and anomalous, this way guaranteeing the integrity of your build process.
Figure 2 – Detection of unexpected file modification in the build
In general, drift examples can range from unexpected file modification (like in the SolarWinds case) and establishing communication with an unapproved IP (like in the Codecov attack). In some cases, these might be legitimate actions – something that your DevOps teams did. But in other cases, it might be a sign of malicious activity and an attack.- Set up assurance policies. Seeing suspicious activity is only part of the solution. To be able to not only detect but also prevent a potential attack, integrity scanning supports assurance policies. This means that you can block the completion of any new builds that show signs of suspicious activity.
For example, here we create a policy to fail the build if Aqua detects a malicious executable being dropped:
Figure 3 – Creating an assurance policy to fail the build if suspicious activity is detected
Detecting the Codecov attack using integrity scanning
Let’s take a real-world case of Codecov and review how we can detect this attack with pipeline integrity scanning.
In the Codecov incident in April 2021, a bash uploader, a script that is downloaded and executed as part of Codecov’s CI process, was modified to steal credentials from the CI environments of customers using it.
Understanding the Codecov baseline
First, we create a CI pipeline that uses Codecov and add our pipeline enforcer step to the workflow. After running the pipeline, the activity tab in the platform looks like this:
Figure 4 – Activity in the test CI pipeline that uses Codecov
We can see under the “Network” tab, there are requests to uploader.codecov.io. The “Files” and “Containers” tabs are empty, and in the “Processes” tab there is a single process called codecov.
Figure 5 – Codecov process in the test CI pipeline
This is the baseline for our test job – we know exactly which network calls should be invoked during our CI pipeline. In other words, we now know what is Codecov’s impact on our CI pipeline.
Protecting against the Codecov attack
To mimic the attack, we added the command that was executed in the Codecov case and ran our CI pipeline again. We can see a new finding in the “Suspicious Behavior” tab:
Figure 6 – Detecting suspicious behavior in the test CI pipeline
Aqua detected a direct IP address communication — one of the many signatures we use to detect suspicious behavior in the pipeline. Blocking direct IP communication improves your CI security, since it is unlikely that you will communicate directly with an IP address instead of a hostname.
To block such cases of direct IP communication, we need to create a dedicated assurance policy:
Figure 7 – Creating an assurance policy to protect against the Codecov attack
Running the pipeline again will now result in a build failure:
Figure 8 – Build failure in the test CI pipeline
Start protecting your software supply chain today
Pipeline integrity scanning solves one of the industry’s most urgent needs to ensure the integrity of the modern development process. By detecting and stopping drift to the original build through eBPF-based scanning and policies, teams can protect their software from malicious tampering and prevent advanced supply chain attacks.
This latest capability is part of the fully featured Aqua Software Supply Chain Security solution that secures your code, all development infrastructure, and pipeline processes so that you can build and ship innovation faster and more securely.
Get a demo of Aqua’s SSCS solution
