Getting Data In

What is the best timestamp format to use for my custom log to be indexed by Splunk?

ftk
Motivator

What is the best timestamp format to use for my custom log to be indexed by Splunk?

Sensible choices are:

Round-trip pattern  2010-08-06T16:43:04.1291862-04:00
Full pattern        Friday, August 06, 2010 4:47:02 PM
RFC1123 pattern     Fri, 06 Aug 2010 16:45:17 GMT
ISO 8601 pattern    2010-08-06T16:45:47
UTC sortable        2010-08-06 16:46:36Z
Tags (1)
1 Solution

gkanapathy
Splunk Employee
Splunk Employee

I tend to prefer the Round-trip, ISO, or UTC sortable pattern or ISO pattern with the TZ info. It doesn't matter too much if you specify an explicit format in Splunk, but it's probably best to indicate a 4-digit year, 2-digit 24-hour hours, numeric (rather than locale-specific textually named) months, and an absolutely unambiguous time zone (e.g., EST is not a good TZ). Day of the week is superfluous. Using GMT/UTC/Zulu time helps to avoid errors and problems around DST switches as well, since that zone never switches.

View solution in original post

vliggio
Communicator

I know this question is ancient, but it can stand using an update.

Splunk is a time based indexing system, so having correct timestamps is critical. From Splunk's logging best practices ( http://dev.splunk.com/view/logging/SP-CAAAFCK 😞

  • Use the most verbose time granularity possible.
  • Put the timestamp at the beginning of the line. The farther you place a timestamp from the beginning, the more difficult it is to tell it's a timestamp and not other data.
  • Include a four-digit year.
  • Include a time zone, preferably a GMT/UTC offset.
  • Time should be rendered in microseconds in each event. The event could become detached from its original source file at some point, so having the most accurate data about an event is ideal.

I would say the "best" custom timestamp that meets all those requirements is an RFC3339 compliant timestamp (see https://tools.ietf.org/html/rfc3339 ). For example:

2017-06-27T14:59:54.158950-04:00.

The strptime (TIME_FORMAT for Splunk) would be %Y-%m-%dT%T.%6N%:z for that timestamp.

splunkreal
Motivator

The link has a problem, copy and paste it : http://dev.splunk.com/view/logging/SP-CAAAFCK

* If this helps, please upvote or accept solution 🙂 *
0 Karma

vliggio
Communicator

Ah, bad html link interpreter - I edited the comment to put a space before the ). Thanks!

0 Karma

ftk
Motivator

Also, I am using Reound-trip pattern mostly now, Splunk parses it easily and it is easy to do in PowerShell (get-date -f s).

0 Karma

ftk
Motivator

Couldn't tell ya buddy, couldn't tell ya 🙂

0 Karma

Lowell
Super Champion

Great question. +1 Why has no one else voted this up?

0 Karma

gkanapathy
Splunk Employee
Splunk Employee

I tend to prefer the Round-trip, ISO, or UTC sortable pattern or ISO pattern with the TZ info. It doesn't matter too much if you specify an explicit format in Splunk, but it's probably best to indicate a 4-digit year, 2-digit 24-hour hours, numeric (rather than locale-specific textually named) months, and an absolutely unambiguous time zone (e.g., EST is not a good TZ). Day of the week is superfluous. Using GMT/UTC/Zulu time helps to avoid errors and problems around DST switches as well, since that zone never switches.

Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...