eBPF technology is seeing strong growth, being widely adopted in the cloud native ecosystem for monitoring, networking, and security goals. At Aqua, along with being used in commercial products, eBPF powers our open source project Tracee to detect events in running containers.
Recently, we released Tracee version 0.6.0, which brings many new features, improves Tracee-eBPF’s event collection, and makes the tool more portable. With these enhanced capabilities, now is the perfect time to reflect on the journey of Tracee, from a tool that collects events in containers to a robust open source runtime security and forensics tool for Linux.
Tracee is born to help collect events in containers
In September 2019, Aqua Security released Tracee to the community as an open source project that provides observability on events in running containers. We decided to open source Tracee so that practitioners and researchers could benefit from its capabilities, while we could gain insights from the community to improve the tool. But the story of Tracee begins before that.
Initially, we built Tracee as an experimental tool for our own needs at Team Nautilus to collect events in running containers. Our goal was to develop a powerful tool that was designed from the ground up for security.
But when you build something from scratch, you need to take one step at a time to develop it into a holistic security tool. Therefore, Tracee’s first version was focused on event collection, such as syscalls (tracee-ebpf). However, we knew that this approach had some limitations. For example, syscall tracing is susceptible to race condition attacks.
Adding more features
As time went by, we got a lot of positive feedback about the tool from the community. We decided to invest further resources and efforts to transform Tracee into a tool suitable for secure tracing.
To achieve this goal, we developed more robust techniques which we first used internally. Once these features matured, we officially added them to Tracee and shared them with the community.
For instance, Linux Security Module (LSM) hooks collection has been available since day one. Since tracing syscalls with Tracee is susceptible to time-of-check, time-of-use (TOCTOU) attacks, LSM hooks can provide more comprehensive and reliable information. We wrote a detailed blog about the challenges we encountered and how we overcame them with LSM hooks.
On top of this, we developed other capabilities, such as uncovering malware payload execution, using Tracee to secure your CI/CD pipelines, and more.
Evolving into a runtime security solution
In February 2021, we released version 0.5.0 of Tracee. This release marked the beginning of Tracee’s evolution from a system tracing CLI tool into a runtime security solution with behavioral analysis capabilities. Tracee now contains a couple of subprojects:
- Tracee-eBPF: a tracing CLI tool that already existed, now renamed
- Tracee-rules: a rule engine to process Tracee-eBPF’s events and detect suspicious behavior based on built-in and user-defined signatures
Tracee-rules, a runtime security detection engine, was designed to help practitioners detect suspicious behavioral patterns in streams of Tracee-eBPF events. Tracee-rules consumes inputs from Tracee-eBPF, processes them with a behavioral pattern signatures engine written in Golang or Rego (OPA), and communicates detections by printing stdout, posting to a webhook, or integrating with external systems.
The next step is to create a library of behavioral signature detections, which we are working on. Currently, the following signatures are available:
| Name | Description | Tags |
| Standard input/output over socket | Redirection of process’s standard input/output to socket | “linux”, “container” |
| Anti-debugging | Process uses anti-debugging technique to block debugger | “linux”, “container” |
| Code injection | Possible code injection into another process | “linux”, “container” |
| Dynamic code loading | Writing to executable allocated memory region | “linux”, “container” |
| Fileless execution | Executing a process from memory, without a file in the disk | “linux”, “container” |
| Kernel module loading | Attempt to load a kernel module detection | “linux”, “container” |
| LD_PRELOAD | Use of LD_PRELOAD to allow hooks on the process | “linux”, “container” |
In August 2021, we released version 0.6.0 of Tracee, which brings the following enhancements:
- Tracee is now more portable thanks to CO:RE. This means that on BTF enabled kernels, you no longer need to compile the eBPF probe or provide kernel-headers. For details, check out the documentation on eBPF compilation.
- Tracee-eBPF can now capture network activity. For more information, see the updated documentation on capturing artifacts.
- New and improved Tracee-eBPF events:
sched_process_exec
security_kernel_read_file
magic_write
Wrap-up
Over the past few years, Tracee has greatly evolved, adding more robust and advanced capabilities. We are proud to offer Tracee to the community as an open source runtime security and forensics tool for Linux, built to address common Linux security issues.
If you are already using Tracee, keep an eye out for new features on the project’s GitHub page.
Watch Team Nautilus’ talks at DEFCON 2021 and Black Hat 2021 to learn how you can use Tracee to detect advanced cyberattacks.