Managing a growing number of systems and applications can be complicated and time consuming, making it particularly important to find a way to centralize and optimize your data. As a result, AWS has recently released FireLens which, working with Fluentd and Fluent Bit, allows you to route your logs to a large number of AWS and third-party destinations using simple configurations in your ECS Task Definition.In this blog we’re providing a step by step guide on forwarding Aqua’s audit logs to Fluent Bit via rsyslog, and then ship the logs to Amazon CloudWatch.
Prerequisites:
1. Ecs-cli installed and configured – https://github.com/aws/amazon-ecs-cli
2. Awscli installed and configured – https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-install.html
Step 1: Create an ecs cluster
ecs-cli configure --cluster aqua-demo --region us-east-1 --config-name aqua-demo
ecs-cli up
--size 2
--instance-type t2.medium
--extra-user-data enable-fluent-log-driver.sh
--keypair aqua_key
--capability-iam
--cluster-config aqua-demo
Step 2: Install Aqua on the cluster
https://github.com/aquasecurity/aqua-aws/tree/master/cloudformation/aqua-ecs
Step 3: Create a fluentbit service that runs as a daemon
We are going to setup an ecs service for fluentbit.
The service is going to listen on port 5140 tcp.
We have the aws-for-fluent-bit image stored in our repo, so we’re going to build the image and push it to the ECR
docker build –tag fluent-bit-demo:0.1
ecs-cli push fluent-bit-demo:0.1
ACCOUNT_ID=$(aws sts get-caller-identity –output text –query ‘Account’)
REGION=<<<YOUR_REGION>>>
aws cloudformation deploy
–template-file ecs-fluent-bit-daemonset.yml
–stack-name ecs-fluent-bit-daemon-service
–parameter-overrides
EnvironmentName=fluentbit-daemon-service
DockerImage=$ACCOUNT_ID.dkr.ecr.$REGION.amazonaws.com/fluent-bit-demo:0.1
Cluster=aqua-demo
–region $(aws configure get region)
–capabilities CAPABILITY_NAMED_IAM
Step 4: Configure aqua to forward logs to fluentbit using syslog
The configuration should look just like the above, though for more robust setups it’s recommended to use service discovery for fluentbit and not ip addresses.
Step 5: Create audit event and find it in CloudWatch
First, we have to setup a policy to block malicious activity, so for this case I have decided that the “uptime” command will be forbidden:
Then we start enforcing the policy on out ECS instances:
Then set the following:
Test time:
As pictured above, the command was blocked.
Optimizing your data is not always a priority need, but as applications grow and their architecture becomes more complicated, it will be important to ensure that you can leverage an easy extension point for streaming logs from containerized applications, either for real-time or retrospective analytics.
Next Steps
Aqua’s integration with AWS FireLens, in addition to the multitude of AWS service integrations across the application lifecycle, enables developers to build, manage and monitor their applications quickly and securely. To learn more about Aqua’s solution, read our whitepaper on the Full Lifecycle Approach to Securing Cloud Native Applications on AWS.