Splunk Cloud Platform

How to configure aws ecs to splunk?

nex6jub
New Member

we are using AWS ECS with fargate and trying to siphon out the container logs to out splunk cloud instance using fluentd. 

1. on the aws ecs side, below is the task_definition.json to create services in ECS -

 

 

{
  "family": "sample-springboot-ms-app",
  "taskRoleArn": "arn:aws:iam::958993399264:role/ecs-task-role",
  "executionRoleArn": "arn:aws:iam::958993399264:role/ecsTaskExecutionRole",
  "networkMode": "awsvpc",
  "containerDefinitions": [
      {
          "name": "sample-springboot-ms-app",
          "image": "958993399264.dkr.ecr.us-east-1.amazonaws.com/dev-repository:finance-sample-springboot-ms-v1-0-0-700950146",
          "cpu": 0,
          "portMappings": [
              {
                  "containerPort": 8080,
                  "hostPort": 8080,
                  "protocol": "tcp"
              }
          ],
          "essential": true,
          "entryPoint": [],
          "command": [],
          "environment": [
              {
                  "name": "APP_CONFIG_VALUE",
                  "value": "12"
              },
              {
                  "name": "START_UP_DELAY",
                  "value": "9"
              },
              {
                  "name": "SIMPLE_TEST",
                  "value": "sample-test-value"
              }
          ],
          "environmentFiles": [],
          "mountPoints": [],
          "volumesFrom": [],
          "secrets": [],
          "logConfiguration": {
            "logDriver": "awsfirelens"
          }
      },
      {
      "logConfiguration": {
        "logDriver": "awslogs",
        "options": {
          "awslogs-group": "debaspreet-debug-fluentd",
          "awslogs-region": "us-east-1",
          "awslogs-stream-prefix": "splunk-ecs"
        }
      },
      "image": "958993399264.dkr.ecr.us-east-1.amazonaws.com/dev-repository:fluent-701086531",
      "firelensConfiguration": {
        "type": "fluentd",
        "options": {
          "config-file-type": "file",
          "config-file-value": "/fluent.conf"
        }
      },
      "essential": true,
      "name": "log_router",
      "memory": 256,
      "memoryReservation": 128
    }
  ],
  "requiresCompatibilities":
  [
      "FARGATE"
  ],
  "cpu": "1024",
  "memory": "2048",
  "runtimePlatform":
  {
      "operatingSystemFamily": "LINUX"
  }
}

 

 

 

 

2. on the fluentd side, below is the fluent.conf -

 

 

<system>
  log_level info
</system>

<match **>
  @type splunk_hec
  protocol https
  hec_host ****************
  hec_port 8088
  hec_token *****************
  index debaspreet
  host_key ec2_instance_id
  source_key ecs_cluster
  sourcetype_key ecs_task_definition
  insecure_ssl true
  <fields>
    container_id
    container_name
    ecs_task_arn
    source
  </fields>
  <format>
    @type single_value
    message_key log
    add_newline false
  </format>
</match>

 

 

 


3. below is the docker file for our custom fluend image that we host in ECR -

 

 

FROM splunk/fluentd-hec:1.2.0
ADD fluent.conf /fluent.conf

 

 


Despite of the above configs, we still dont see the container logs in splunk. Not sure whats incorrect in the config or whats missing. Out splunk cloud instance has been setup correctly because we do see the below post message there -

 

 

curl -k  https://****************.com:8088/services/collector/event -H "Authorization: Splunk ****************" -d '{"event": "hello world"}'

 

 


Any pointers as to why this config isnt working ?
Thanks

 

Labels (1)
Tags (1)
0 Karma
Get Updates on the Splunk Community!

Thanks for the Memories! Splunk University, .conf24, and Community Connections

Thank you to everyone in the Splunk Community who joined us for .conf24 – starting with Splunk University and ...

.conf24 | Day 0

Hello Splunk Community! My name is Chris, and I'm based in Canberra, Australia's capital, and I travelled for ...

Enhance Security Visibility with Splunk Enterprise Security 7.1 through Threat ...

 (view in My Videos)Struggling with alert fatigue, lack of context, and prioritization around security ...