topic Re: What is the best timestamp format to use for my custom log to be indexed by Splunk? in Getting Data In

What is the best timestamp format to use for my custom log to be indexed by Splunk?

ftk — Sat, 07 Aug 2010 03:49:27 GMT

What is the best timestamp format to use for my custom log to be indexed by Splunk?

Sensible choices are:

Round-trip pattern  2010-08-06T16:43:04.1291862-04:00
Full pattern        Friday, August 06, 2010 4:47:02 PM
RFC1123 pattern     Fri, 06 Aug 2010 16:45:17 GMT
ISO 8601 pattern    2010-08-06T16:45:47
UTC sortable        2010-08-06 16:46:36Z

Re: What is the best timestamp format to use for my custom log to be indexed by Splunk?

gkanapathy — Sat, 07 Aug 2010 04:41:22 GMT

I tend to prefer the Round-trip, ISO, or UTC sortable pattern or ISO pattern with the TZ info. It doesn't matter too much if you specify an explicit format in Splunk, but it's probably best to indicate a 4-digit year, 2-digit 24-hour hours, numeric (rather than locale-specific textually named) months, and an absolutely unambiguous time zone (e.g., EST is not a good TZ). Day of the week is superfluous. Using GMT/UTC/Zulu time helps to avoid errors and problems around DST switches as well, since that zone never switches.

Re: What is the best timestamp format to use for my custom log to be indexed by Splunk?

Lowell — Sat, 19 Feb 2011 01:44:49 GMT

Great question. +1 Why has no one else voted this up?

Re: What is the best timestamp format to use for my custom log to be indexed by Splunk?

ftk — Sat, 19 Feb 2011 06:06:00 GMT

Couldn't tell ya buddy, couldn't tell ya 🙂

Re: What is the best timestamp format to use for my custom log to be indexed by Splunk?

ftk — Sat, 19 Feb 2011 06:07:46 GMT

Also, I am using Reound-trip pattern mostly now, Splunk parses it easily and it is easy to do in PowerShell (get-date -f s).

Re: What is the best timestamp format to use for my custom log to be indexed by Splunk?

vliggio — Tue, 27 Jun 2017 19:16:11 GMT

I know this question is ancient, but it can stand using an update.

Splunk is a time based indexing system, so having correct timestamps is critical. From Splunk's logging best practices ( http://dev.splunk.com/view/logging/SP-CAAAFCK 😞

Use the most verbose time granularity possible.
Put the timestamp at the beginning of the line. The farther you place a timestamp from the beginning, the more difficult it is to tell it's a timestamp and not other data.
Include a four-digit year.
Include a time zone, preferably a GMT/UTC offset.
Time should be rendered in microseconds in each event. The event could become detached from its original source file at some point, so having the most accurate data about an event is ideal.

I would say the "best" custom timestamp that meets all those requirements is an RFC3339 compliant timestamp (see https://tools.ietf.org/html/rfc3339 ). For example:

2017-06-27T14:59:54.158950-04:00.

The strptime (TIME_FORMAT for Splunk) would be %Y-%m-%dT%T.%6N%:z for that timestamp.

Re: What is the best timestamp format to use for my custom log to be indexed by Splunk?

splunkreal — Wed, 03 Jan 2018 11:04:13 GMT

The link has a problem, copy and paste it : http://dev.splunk.com/view/logging/SP-CAAAFCK

Re: What is the best timestamp format to use for my custom log to be indexed by Splunk?

vliggio — Wed, 03 Jan 2018 14:37:44 GMT

Ah, bad html link interpreter - I edited the comment to put a space before the ). Thanks!