Log format and ingestion in Splunk

[tayshawn]

Hello everyone! We have a container service running on AWS ECS with Splunk log driver enabled (via HEC token). 

At moment, we found log lines look awful (see below example). Also, no event level filtered

{ [-]
   line: xxxxxxxxx - - [16/Sep/2023:23:59:59 +0000] "GET /health HTTP/1.1" 200 236 "-" "ELB-HealthChecker/2.0" "-" 
   source: stdout 
   tag: xxxxxxxxxxx 
} Show as raw text

host = xxx source = xxx source = xxx sourcetype = xxxx

We would like to make changes in Splunk to ensure the events are in a better-formatted standard as following:

Sep 19 03:27:09 ip-xxx.xxxx xx[16151]: xxx ERROR xx - DIST:xx.xx BAS:8  NID:w-xxxxxx RID:b FID:bxxxx WSID:xxxx 

host = xxx level = ERROR source = xxx sourcetype =  xxx

 We do have log forwarder rule configured (logs for other services are all formatted as above) . May I get some helps to reformat logs? Much appreciated! 

[gcusello]

Hi @tayshawn,

this isn't a Splunk question:

if you AWS ECS sends logs in json format, you should ask to AWS if it's possible to have logs in a different format, but probably it's very difficoult!

Anyway, if you use the Splunk Add-On for AWS, you should have the parser to read these logs and extract all the fields, so you can put them in a table as you want, but without changing the original source.

Ciao.

Giuseppe