Splunk Search

Log format and ingestion in Splunk

tayshawn
New Member

Hello everyone! We have a container service running on AWS ECS with Splunk log driver enabled (via HEC token). 

At moment, we found log lines look awful (see below example). Also, no event level filtered

{ [-]
   line: xxxxxxxxx - - [16/Sep/2023:23:59:59 +0000] "GET /health HTTP/1.1" 200 236 "-" "ELB-HealthChecker/2.0" "-"
   source: stdout
   tag: xxxxxxxxxxx

} Show as raw text

host = xxx source = xxx source = xxx sourcetype = xxxx

 

We would like to make changes in Splunk to ensure the events are in a better-formatted standard as following:

Sep 19 03:27:09 ip-xxx.xxxx xx[16151]: xxx ERROR xx - DIST:xx.xx BAS:8  NID:w-xxxxxx RID:b FID:bxxxx WSID:xxxx 

host = xxx level = ERROR source = xxx sourcetype =  xxx

 We do have log forwarder rule configured (logs for other services are all formatted as above) . May I get some helps to reformat logs? Much appreciated! 

Labels (2)
0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @tayshawn,

this isn't a Splunk question:

if you AWS ECS sends logs in json format, you should ask to AWS if it's possible to have logs in a different format, but probably it's very difficoult!

Anyway, if you use the Splunk Add-On for AWS, you should have the parser to read these logs and extract all the fields, so you can put them in a table as you want, but without changing the original source.

Ciao.

Giuseppe

0 Karma
Get Updates on the Splunk Community!

Splunk Observability Cloud's AI Assistant in Action Series: Auditing Compliance and ...

This is the third post in the Splunk Observability Cloud’s AI Assistant in Action series that digs into how to ...

Splunk Community Badges!

  Hey everyone! Ready to earn some serious bragging rights in the community? Along with our existing badges ...

What You Read The Most: Splunk Lantern’s Most Popular Articles!

Splunk Lantern is a Splunk customer success center that provides advice from Splunk experts on valuable data ...