Splunk Cloud Platform

How to configure aws ecs to splunk?

nex6jub
New Member

we are using AWS ECS with fargate and trying to siphon out the container logs to out splunk cloud instance using fluentd. 

1. on the aws ecs side, below is the task_definition.json to create services in ECS -

 

 

{
  "family": "sample-springboot-ms-app",
  "taskRoleArn": "arn:aws:iam::958993399264:role/ecs-task-role",
  "executionRoleArn": "arn:aws:iam::958993399264:role/ecsTaskExecutionRole",
  "networkMode": "awsvpc",
  "containerDefinitions": [
      {
          "name": "sample-springboot-ms-app",
          "image": "958993399264.dkr.ecr.us-east-1.amazonaws.com/dev-repository:finance-sample-springboot-ms-v1-0-0-700950146",
          "cpu": 0,
          "portMappings": [
              {
                  "containerPort": 8080,
                  "hostPort": 8080,
                  "protocol": "tcp"
              }
          ],
          "essential": true,
          "entryPoint": [],
          "command": [],
          "environment": [
              {
                  "name": "APP_CONFIG_VALUE",
                  "value": "12"
              },
              {
                  "name": "START_UP_DELAY",
                  "value": "9"
              },
              {
                  "name": "SIMPLE_TEST",
                  "value": "sample-test-value"
              }
          ],
          "environmentFiles": [],
          "mountPoints": [],
          "volumesFrom": [],
          "secrets": [],
          "logConfiguration": {
            "logDriver": "awsfirelens"
          }
      },
      {
      "logConfiguration": {
        "logDriver": "awslogs",
        "options": {
          "awslogs-group": "debaspreet-debug-fluentd",
          "awslogs-region": "us-east-1",
          "awslogs-stream-prefix": "splunk-ecs"
        }
      },
      "image": "958993399264.dkr.ecr.us-east-1.amazonaws.com/dev-repository:fluent-701086531",
      "firelensConfiguration": {
        "type": "fluentd",
        "options": {
          "config-file-type": "file",
          "config-file-value": "/fluent.conf"
        }
      },
      "essential": true,
      "name": "log_router",
      "memory": 256,
      "memoryReservation": 128
    }
  ],
  "requiresCompatibilities":
  [
      "FARGATE"
  ],
  "cpu": "1024",
  "memory": "2048",
  "runtimePlatform":
  {
      "operatingSystemFamily": "LINUX"
  }
}

 

 

 

 

2. on the fluentd side, below is the fluent.conf -

 

 

<system>
  log_level info
</system>

<match **>
  @type splunk_hec
  protocol https
  hec_host ****************
  hec_port 8088
  hec_token *****************
  index debaspreet
  host_key ec2_instance_id
  source_key ecs_cluster
  sourcetype_key ecs_task_definition
  insecure_ssl true
  <fields>
    container_id
    container_name
    ecs_task_arn
    source
  </fields>
  <format>
    @type single_value
    message_key log
    add_newline false
  </format>
</match>

 

 

 


3. below is the docker file for our custom fluend image that we host in ECR -

 

 

FROM splunk/fluentd-hec:1.2.0
ADD fluent.conf /fluent.conf

 

 


Despite of the above configs, we still dont see the container logs in splunk. Not sure whats incorrect in the config or whats missing. Out splunk cloud instance has been setup correctly because we do see the below post message there -

 

 

curl -k  https://****************.com:8088/services/collector/event -H "Authorization: Splunk ****************" -d '{"event": "hello world"}'

 

 


Any pointers as to why this config isnt working ?
Thanks

 

Labels (1)
Tags (1)
0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...