Optimized Logging for Container Workloads

Optimized Logging for Container Workloads Using AWS FireLens

Managing a growing number of systems and applications can be complicated and time consuming, making it particularly important to find a way to centralize and optimize your data. As a result, AWS has recently released FireLens which, working with Fluentd and Fluent Bit, allows you to route your logs to a large number of AWS and third-party destinations using simple configurations in your ECS Task Definition.In this blog we’re providing a step by step guide on forwarding Aqua’s audit logs to Fluent Bit via rsyslog, and then ship the logs to Amazon CloudWatch.

Prerequisites:

1. Ecs-cli installed and configured - https://github.com/aws/amazon-ecs-cli
2. Awscli installed and configured - https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-install.html

Step 1: Create an ecs cluster

ecs-cli configure --cluster aqua-demo --region us-east-1 --config-name aqua-demo

ecs-cli up \
     --size 2 \
     --instance-type t2.medium \
     --extra-user-data enable-fluent-log-driver.sh \
     --keypair aqua_key \
     --capability-iam \
     --cluster-config aqua-demo

Step 2: Install Aqua on the cluster

https://github.com/aquasecurity/aqua-aws/tree/master/cloudformation/aqua-ecs

Step 3: Create a fluentbit service that runs as a daemon

We are going to setup an ecs service for fluentbit.
The service is going to listen on port 5140 tcp.
We have the aws-for-fluent-bit image stored in our repo, so we’re going to build the image and push it to the ECR

docker build --tag fluent-bit-demo:0.1
ecs-cli push fluent-bit-demo:0.1

ACCOUNT_ID=$(aws sts get-caller-identity --output text --query 'Account')
REGION=<<<YOUR_REGION>>>

aws cloudformation deploy \
      --template-file ecs-fluent-bit-daemonset.yml \
      --stack-name ecs-fluent-bit-daemon-service \
      --parameter-overrides \
      EnvironmentName=fluentbit-daemon-service \
      DockerImage=$ACCOUNT_ID.dkr.ecr.$REGION.amazonaws.com/fluent-bit-demo:0.1 \
      Cluster=aqua-demo \
      --region $(aws configure get region) \
      --capabilities CAPABILITY_NAMED_IAM

Step 4: Configure aqua to forward logs to fluentbit using syslog

image 1

image 2

The configuration should look just like the above, though for more robust setups it’s recommended to use service discovery for fluentbit and not ip addresses.

Step 5: Create audit event and find it in CloudWatch

First, we have to setup a policy to block malicious activity, so for this case I have decided that the “uptime” command will be forbidden:

image 3

Then we start enforcing the policy on out ECS instances:

image 4

Then set the following:

image 5

image 6

Test time:

image 7

As pictured above, the command was blocked.

Optimizing your data is not always a priority need, but as applications grow and their architecture becomes more complicated, it will be important to ensure that you can leverage an easy extension point for streaming logs from containerized applications, either for real-time or retrospective analytics.

Next Steps

Aqua’s integration with AWS FireLens, in addition to the multitude of AWS service integrations across the application lifecycle, enables developers to build, manage and monitor their applications quickly and securely. To learn more about Aqua’s solution, read our whitepaper on the Full Lifecycle Approach to Securing Cloud Native Applications on AWS.

Try Aqua AWS Marketplace

Picture of Reshef Sharvit

Reshef Sharvit

Reshef is DevOps Engineer at Aqua Security.

Container Security, AWS Security, Cloud Native Security