Splunk Cloud Platform

How to configure aws ecs to splunk?

nex6jub
New Member

we are using AWS ECS with fargate and trying to siphon out the container logs to out splunk cloud instance using fluentd. 

1. on the aws ecs side, below is the task_definition.json to create services in ECS -

 

 

{
  "family": "sample-springboot-ms-app",
  "taskRoleArn": "arn:aws:iam::958993399264:role/ecs-task-role",
  "executionRoleArn": "arn:aws:iam::958993399264:role/ecsTaskExecutionRole",
  "networkMode": "awsvpc",
  "containerDefinitions": [
      {
          "name": "sample-springboot-ms-app",
          "image": "958993399264.dkr.ecr.us-east-1.amazonaws.com/dev-repository:finance-sample-springboot-ms-v1-0-0-700950146",
          "cpu": 0,
          "portMappings": [
              {
                  "containerPort": 8080,
                  "hostPort": 8080,
                  "protocol": "tcp"
              }
          ],
          "essential": true,
          "entryPoint": [],
          "command": [],
          "environment": [
              {
                  "name": "APP_CONFIG_VALUE",
                  "value": "12"
              },
              {
                  "name": "START_UP_DELAY",
                  "value": "9"
              },
              {
                  "name": "SIMPLE_TEST",
                  "value": "sample-test-value"
              }
          ],
          "environmentFiles": [],
          "mountPoints": [],
          "volumesFrom": [],
          "secrets": [],
          "logConfiguration": {
            "logDriver": "awsfirelens"
          }
      },
      {
      "logConfiguration": {
        "logDriver": "awslogs",
        "options": {
          "awslogs-group": "debaspreet-debug-fluentd",
          "awslogs-region": "us-east-1",
          "awslogs-stream-prefix": "splunk-ecs"
        }
      },
      "image": "958993399264.dkr.ecr.us-east-1.amazonaws.com/dev-repository:fluent-701086531",
      "firelensConfiguration": {
        "type": "fluentd",
        "options": {
          "config-file-type": "file",
          "config-file-value": "/fluent.conf"
        }
      },
      "essential": true,
      "name": "log_router",
      "memory": 256,
      "memoryReservation": 128
    }
  ],
  "requiresCompatibilities":
  [
      "FARGATE"
  ],
  "cpu": "1024",
  "memory": "2048",
  "runtimePlatform":
  {
      "operatingSystemFamily": "LINUX"
  }
}

 

 

 

 

2. on the fluentd side, below is the fluent.conf -

 

 

<system>
  log_level info
</system>

<match **>
  @type splunk_hec
  protocol https
  hec_host ****************
  hec_port 8088
  hec_token *****************
  index debaspreet
  host_key ec2_instance_id
  source_key ecs_cluster
  sourcetype_key ecs_task_definition
  insecure_ssl true
  <fields>
    container_id
    container_name
    ecs_task_arn
    source
  </fields>
  <format>
    @type single_value
    message_key log
    add_newline false
  </format>
</match>

 

 

 


3. below is the docker file for our custom fluend image that we host in ECR -

 

 

FROM splunk/fluentd-hec:1.2.0
ADD fluent.conf /fluent.conf

 

 


Despite of the above configs, we still dont see the container logs in splunk. Not sure whats incorrect in the config or whats missing. Out splunk cloud instance has been setup correctly because we do see the below post message there -

 

 

curl -k  https://****************.com:8088/services/collector/event -H "Authorization: Splunk ****************" -d '{"event": "hello world"}'

 

 


Any pointers as to why this config isnt working ?
Thanks

 

Labels (1)
Tags (1)
0 Karma
Get Updates on the Splunk Community!

Index This | I’m short for "configuration file.” What am I?

May 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with a Special ...

New Articles from Academic Learning Partners, Help Expand Lantern’s Use Case Library, ...

Splunk Lantern is a Splunk customer success center that provides advice from Splunk experts on valuable data ...

Your Guide to SPL2 at .conf24!

So, you’re headed to .conf24? You’re in for a good time. Las Vegas weather is just *chef’s kiss* beautiful in ...