Getting Data In

timestamp recognition for elb logs

jblphx
Engager

I am trying to input ELB logs that are being provided to me in the following format:

elb-instance-hostname [02/10/2012:17:59:37.430204 -0000] 23.22.69.42:42455 10.158.21.55:8081 0.000046/0.148519/0.000025 200 200 0 3 "GET https://elb-instance-hostname.com/api/v1/assets/d2d9d161-1ca9-412f-a98f-4ce4049a8ee5/conversions HTTP/1.1"

Splunk is having a hard time automatically recognizing the timestamp, and as such is storing the events in the index with the incorrect date.

How would the TIME_FORMAT need to look for these events, to properly parse the timestamps?

Tags (1)

emiller42
Motivator

I would try the following in your props.conf stanza for this sourcetype:

TIME_FORMAT = %m/%d/%Y:%H:%M:%S.%6 %z

You can also give it some more flags to narrow down the scope of where it looks for timestamps:

TIME_PREFIX=\[

(The first '[' seen indicates that the timestamp starts on the next character)

MAX_TIMESTAMP_LOOKAHEAD  = 32

(The max number of characters it will scan past the time_prefix to extract the timestamp. Handy for performance reasons if some events are exceptionally long. In this case, your timestamp is always 32 characters long so you can get specific here)

I also find the following helpful when building strptime: strftime.net

Get Updates on the Splunk Community!

.conf24 | Day 0

Hello Splunk Community! My name is Chris, and I'm based in Canberra, Australia's capital, and I travelled for ...

Enhance Security Visibility with Splunk Enterprise Security 7.1 through Threat ...

 (view in My Videos)Struggling with alert fatigue, lack of context, and prioritization around security ...

Troubleshooting the OpenTelemetry Collector

  In this tech talk, you’ll learn how to troubleshoot the OpenTelemetry collector - from checking the ...