we are using AWS ECS with fargate and trying to siphon out the container logs to out splunk cloud instance using fluentd.
1. on the aws ecs side, below is the task_definition.json to create services in ECS -
{
"family": "sample-springboot-ms-app",
"taskRoleArn": "arn:aws:iam::958993399264:role/ecs-task-role",
"executionRoleArn": "arn:aws:iam::958993399264:role/ecsTaskExecutionRole",
"networkMode": "awsvpc",
"containerDefinitions": [
{
"name": "sample-springboot-ms-app",
"image": "958993399264.dkr.ecr.us-east-1.amazonaws.com/dev-repository:finance-sample-springboot-ms-v1-0-0-700950146",
"cpu": 0,
"portMappings": [
{
"containerPort": 8080,
"hostPort": 8080,
"protocol": "tcp"
}
],
"essential": true,
"entryPoint": [],
"command": [],
"environment": [
{
"name": "APP_CONFIG_VALUE",
"value": "12"
},
{
"name": "START_UP_DELAY",
"value": "9"
},
{
"name": "SIMPLE_TEST",
"value": "sample-test-value"
}
],
"environmentFiles": [],
"mountPoints": [],
"volumesFrom": [],
"secrets": [],
"logConfiguration": {
"logDriver": "awsfirelens"
}
},
{
"logConfiguration": {
"logDriver": "awslogs",
"options": {
"awslogs-group": "debaspreet-debug-fluentd",
"awslogs-region": "us-east-1",
"awslogs-stream-prefix": "splunk-ecs"
}
},
"image": "958993399264.dkr.ecr.us-east-1.amazonaws.com/dev-repository:fluent-701086531",
"firelensConfiguration": {
"type": "fluentd",
"options": {
"config-file-type": "file",
"config-file-value": "/fluent.conf"
}
},
"essential": true,
"name": "log_router",
"memory": 256,
"memoryReservation": 128
}
],
"requiresCompatibilities":
[
"FARGATE"
],
"cpu": "1024",
"memory": "2048",
"runtimePlatform":
{
"operatingSystemFamily": "LINUX"
}
}
2. on the fluentd side, below is the fluent.conf -
<system>
log_level info
</system>
<match **>
@type splunk_hec
protocol https
hec_host ****************
hec_port 8088
hec_token *****************
index debaspreet
host_key ec2_instance_id
source_key ecs_cluster
sourcetype_key ecs_task_definition
insecure_ssl true
<fields>
container_id
container_name
ecs_task_arn
source
</fields>
<format>
@type single_value
message_key log
add_newline false
</format>
</match>
3. below is the docker file for our custom fluend image that we host in ECR -
FROM splunk/fluentd-hec:1.2.0
ADD fluent.conf /fluent.conf
Despite of the above configs, we still dont see the container logs in splunk. Not sure whats incorrect in the config or whats missing. Out splunk cloud instance has been setup correctly because we do see the below post message there -
curl -k https://****************.com:8088/services/collector/event -H "Authorization: Splunk ****************" -d '{"event": "hello world"}'
Any pointers as to why this config isnt working ?
Thanks