Getting Data In

What is the best timestamp format to use for my custom log to be indexed by Splunk?

ftk
Motivator

What is the best timestamp format to use for my custom log to be indexed by Splunk?

Sensible choices are:

Round-trip pattern  2010-08-06T16:43:04.1291862-04:00
Full pattern        Friday, August 06, 2010 4:47:02 PM
RFC1123 pattern     Fri, 06 Aug 2010 16:45:17 GMT
ISO 8601 pattern    2010-08-06T16:45:47
UTC sortable        2010-08-06 16:46:36Z
Tags (1)
1 Solution

gkanapathy
Splunk Employee
Splunk Employee

I tend to prefer the Round-trip, ISO, or UTC sortable pattern or ISO pattern with the TZ info. It doesn't matter too much if you specify an explicit format in Splunk, but it's probably best to indicate a 4-digit year, 2-digit 24-hour hours, numeric (rather than locale-specific textually named) months, and an absolutely unambiguous time zone (e.g., EST is not a good TZ). Day of the week is superfluous. Using GMT/UTC/Zulu time helps to avoid errors and problems around DST switches as well, since that zone never switches.

View solution in original post

vliggio
Communicator

I know this question is ancient, but it can stand using an update.

Splunk is a time based indexing system, so having correct timestamps is critical. From Splunk's logging best practices ( http://dev.splunk.com/view/logging/SP-CAAAFCK 😞

  • Use the most verbose time granularity possible.
  • Put the timestamp at the beginning of the line. The farther you place a timestamp from the beginning, the more difficult it is to tell it's a timestamp and not other data.
  • Include a four-digit year.
  • Include a time zone, preferably a GMT/UTC offset.
  • Time should be rendered in microseconds in each event. The event could become detached from its original source file at some point, so having the most accurate data about an event is ideal.

I would say the "best" custom timestamp that meets all those requirements is an RFC3339 compliant timestamp (see https://tools.ietf.org/html/rfc3339 ). For example:

2017-06-27T14:59:54.158950-04:00.

The strptime (TIME_FORMAT for Splunk) would be %Y-%m-%dT%T.%6N%:z for that timestamp.

splunkreal
Motivator

The link has a problem, copy and paste it : http://dev.splunk.com/view/logging/SP-CAAAFCK

* If this helps, please upvote or accept solution 🙂 *
0 Karma

vliggio
Communicator

Ah, bad html link interpreter - I edited the comment to put a space before the ). Thanks!

0 Karma

ftk
Motivator

Also, I am using Reound-trip pattern mostly now, Splunk parses it easily and it is easy to do in PowerShell (get-date -f s).

0 Karma

ftk
Motivator

Couldn't tell ya buddy, couldn't tell ya 🙂

0 Karma

Lowell
Super Champion

Great question. +1 Why has no one else voted this up?

0 Karma

gkanapathy
Splunk Employee
Splunk Employee

I tend to prefer the Round-trip, ISO, or UTC sortable pattern or ISO pattern with the TZ info. It doesn't matter too much if you specify an explicit format in Splunk, but it's probably best to indicate a 4-digit year, 2-digit 24-hour hours, numeric (rather than locale-specific textually named) months, and an absolutely unambiguous time zone (e.g., EST is not a good TZ). Day of the week is superfluous. Using GMT/UTC/Zulu time helps to avoid errors and problems around DST switches as well, since that zone never switches.

Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

Splunk is officially part of Cisco

Revolutionizing how our customers build resilience across their entire digital footprint.   Splunk ...

Splunk APM & RUM | Planned Maintenance March 26 - March 28, 2024

There will be planned maintenance for Splunk APM and RUM between March 26, 2024 and March 28, 2024 as ...