Shipping Logs from AWS CloudWatch to ElasticSearch with Fargate and Filebeat

Centralizing logs and efficiently analyzing them is vital for maintaining the health of your system. In this guide, we’ll walk through how to ship logs from AWS CloudWatch to ElasticSearch using Filebeat running on a Fargate cluster.

Requirements

  • A Fargate Cluster.
  • IAM roles for task execution and access control.
  • Filebeat configuration.

Setting Up a Fargate Cluster

Before we proceed, you’ll need to have an operational Fargate cluster ready to run the task.

IAM Roles Configuration

IAM roles define what ECS and the running process can do. AWS documents this split between task and execution roles.

  1. Task role: Supplies credentials to Filebeat through ECS task metadata. It—not the execution role—needs permission to read the selected CloudWatch log groups.

    {
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "logs:DescribeLogGroups",
        "logs:GetQueryResults"
      ],
      "Resource": "*"
    },
    {
      "Effect": "Allow",
      "Action": [
        "logs:FilterLogEvents",
        "logs:GetLogEvents",
        "logs:StartQuery"
      ],
      "Resource": "arn:aws:logs:YOUR_REGION:YOUR_ACCOUNT_ID:log-group:YOUR_LOG_GROUP_TO_SHIP:*"
    }
  ]
}

  2. Execution role: Used by the ECS agent to pull the image and send this container’s own logs. It is not the identity Filebeat uses for AWS API calls.

    {
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "ecr:GetAuthorizationToken",
        "ecr:BatchCheckLayerAvailability",
        "ecr:GetDownloadUrlForLayer",
        "ecr:BatchGetImage",
        "logs:CreateLogGroup",
        "logs:CreateLogStream",
        "logs:PutLogEvents"
      ],
      "Resource": "*"
    }
  ]
}

Filebeat Configuration

Filebeat is the tool that will be used to read logs from CloudWatch and forward them to ElasticSearch. Here’s a sample configuration for Filebeat:

Includes settings, input for AWS Cloudwatch, and output to Logstash

name: 'filebeat-cloudwatch'
tags: []
fields_under_root: true
fields:
  domain: 'domain.com'
  cluster: 'my-pony-cluster'

logging.level: debug

# This can be used to debug the input by going to
# :5066/inputs/?pretty <-- this will show the status of the cloudwatch input
http:
  enabled: true
  host: 127.0.0.1
  port: 5066

http.pprof.enabled: false

# Persist the Filebeat registry across task replacements.
path.data: /mnt/share/filebeat-data

filebeat.inputs:
  - type: aws-cloudwatch
    log_group_arn: arn:aws:logs:YOUR_REGION:YOUR_ACCOUND_ID:log-group:YOUR_LOG_GROUP_TO_SHIP:*
    scan_frequency: 1m
    start_position: end
    region_name: YOUR_REGION
    fields_under_root: true
    fields:
      logzio_codec: json
      type: fargate

processors:
  - rename:
      ignore_missing: true
      fields:
        - from: 'agent'
          to: 'beat_agent'
        - from: 'log.file.path'
          to: 'source'

output:
  logstash:
    hosts:
      - 'logstash:5015'
    ssl:
      certificate_authorities:
        - |
          -----BEGIN CERTIFICATE-----
          ...
          ...
          -----END CERTIFICATE-----

Fargate Service Configuration

Define a Fargate service and task in JSON format. These will include the details about the cluster, desired count of running tasks, network configurations, and container definitions.

JSON for Fargate Service

{
  "cluster": "MyTestCluster",
  "serviceName": "filebeat",
  "taskDefinition": "arn:aws:ecs:YOUR_REGION:YOUR_ACCOUND_ID:task-definition/filebeat:1",
  "desiredCount": 1,
  "launchType": "FARGATE",
  "deploymentConfiguration": {
    "maximumPercent": 200,
    "minimumHealthyPercent": 0
  },
  "networkConfiguration": {
    "awsvpcConfiguration": {
      "subnets": ["subnet-***********"],
      "securityGroups": ["sg-************"],
      "assignPublicIp": "DISABLED"
    }
  }
}

JSON for Fargate Task

{
  "networkMode": "awsvpc",
  "taskRoleArn": "arn:aws:iam::YOUR_ACCOUND_ID:role/ecsTaskRole",
  "executionRoleArn": "arn:aws:iam::YOUR_ACCOUND_ID:role/ecsTaskExecutionRole",
  "cpu": "256",
  "memory": "512",
  "containerDefinitions": [
    {
      "name": "filebeat",
      "essential": true,
      "image": "docker.elastic.co/beats/filebeat:FILEBEAT_VERSION",
      "command": ["-c", "/mnt/share/filebeat/filebeat.yml"],
      "mountPoints": [
        {
          "sourceVolume": "efs-volume",
          "containerPath": "/mnt/share/"
        }
      ],
      "logConfiguration": {
        "logDriver": "awslogs",
        "options": {
          "awslogs-group": "YOUR_LOG_GROUP_TO_MONITOR_FILEBEAT",
          "awslogs-region": "YOUR_REGION",
          "awslogs-create-group": "true",
          "awslogs-stream-prefix": "filebeat"
        }
      }
    }
  ],
  "volumes": [
    {
      "name": "efs-volume",
      "efsVolumeConfiguration": {
        "fileSystemId": "fs-****************",
        "transitEncryption": "ENABLED"
      }
    }
  ],
  "requiresCompatibilities": ["FARGATE"],
  "family": "filebeat"
}

The logConfiguration block is optional; it sends Filebeat’s own container logs to CloudWatch and is separate from the log groups Filebeat collects.

Deployment

With the configurations ready, you can deploy the Filebeat on the Fargate cluster:

  1. Deploy the task definition.
  2. Deploy the service definition.
  3. Monitor your logs in ElasticSearch.

I replace FILEBEAT_VERSION during task-definition generation with the Filebeat release compatible with Elasticsearch/Logstash instead of leaving the old 8.8.2 image hard-coded. Elastic publishes the compatibility matrix.

The EFS-backed path.data holds Filebeat’s registry. Keeping it prevents a replacement task replaying the whole available CloudWatch window; losing or sharing that state can cause replay or duplicates, so one collector task owns the directory. Filebeat gets short-lived AWS credentials from its task role—there are no static access keys in this definition. Diagnostics stay on loopback and can be inspected through ECS Exec.


Buy Me a Coffee