Getting Data In

What is the best timestamp format to use for my custom log to be indexed by Splunk?

Motivator

What is the best timestamp format to use for my custom log to be indexed by Splunk?

Sensible choices are:

Round-trip pattern  2010-08-06T16:43:04.1291862-04:00
Full pattern        Friday, August 06, 2010 4:47:02 PM
RFC1123 pattern     Fri, 06 Aug 2010 16:45:17 GMT
ISO 8601 pattern    2010-08-06T16:45:47
UTC sortable        2010-08-06 16:46:36Z
Tags (1)
1 Solution

Splunk Employee
Splunk Employee

I tend to prefer the Round-trip, ISO, or UTC sortable pattern or ISO pattern with the TZ info. It doesn't matter too much if you specify an explicit format in Splunk, but it's probably best to indicate a 4-digit year, 2-digit 24-hour hours, numeric (rather than locale-specific textually named) months, and an absolutely unambiguous time zone (e.g., EST is not a good TZ). Day of the week is superfluous. Using GMT/UTC/Zulu time helps to avoid errors and problems around DST switches as well, since that zone never switches.

View solution in original post

Communicator

I know this question is ancient, but it can stand using an update.

Splunk is a time based indexing system, so having correct timestamps is critical. From Splunk's logging best practices ( http://dev.splunk.com/view/logging/SP-CAAAFCK 😞

  • Use the most verbose time granularity possible.
  • Put the timestamp at the beginning of the line. The farther you place a timestamp from the beginning, the more difficult it is to tell it's a timestamp and not other data.
  • Include a four-digit year.
  • Include a time zone, preferably a GMT/UTC offset.
  • Time should be rendered in microseconds in each event. The event could become detached from its original source file at some point, so having the most accurate data about an event is ideal.

I would say the "best" custom timestamp that meets all those requirements is an RFC3339 compliant timestamp (see https://tools.ietf.org/html/rfc3339 ). For example:

2017-06-27T14:59:54.158950-04:00.

The strptime (TIME_FORMAT for Splunk) would be %Y-%m-%dT%T.%6N%:z for that timestamp.

Builder

The link has a problem, copy and paste it : http://dev.splunk.com/view/logging/SP-CAAAFCK

0 Karma

Communicator

Ah, bad html link interpreter - I edited the comment to put a space before the ). Thanks!

0 Karma

Motivator

Also, I am using Reound-trip pattern mostly now, Splunk parses it easily and it is easy to do in PowerShell (get-date -f s).

0 Karma

Motivator

Couldn't tell ya buddy, couldn't tell ya 🙂

0 Karma

Super Champion

Great question. +1 Why has no one else voted this up?

0 Karma

Splunk Employee
Splunk Employee

I tend to prefer the Round-trip, ISO, or UTC sortable pattern or ISO pattern with the TZ info. It doesn't matter too much if you specify an explicit format in Splunk, but it's probably best to indicate a 4-digit year, 2-digit 24-hour hours, numeric (rather than locale-specific textually named) months, and an absolutely unambiguous time zone (e.g., EST is not a good TZ). Day of the week is superfluous. Using GMT/UTC/Zulu time helps to avoid errors and problems around DST switches as well, since that zone never switches.

View solution in original post