Getting Data In
Highlighted

How to parse an unusual timestamp format?

New Member

Hi all,

I have have some inconsistent timestamp parsing issues that I believe are due to an incorrect TIME_FORMAT value in my props.conf file and I am hoping that someone may be able to clarify what I've done wrong here. I get timestamps ingested into my Splunk instance with a format like this:

2017-01-31T19:35:43.379Z

This is the TIME_FORMAT value I have been using:

TIME_FORMAT = %Y-%m-%dT%H:%M:%S.%3N

It is mostly consistent but sometimes appears to not get parsed. Splunk Support has only been able to suggest at the Z at the end might be the issue, and on review of the documentation I don't see a specific way to note that in the TIMEFORMAT string. Does anyone know how to structure TIMEFORMAT to properly capture this?
I am not able to alter this data in any way so I must work with this format as-is.

0 Karma
Highlighted

Re: How to parse an unusual timestamp format?

SplunkTrust
SplunkTrust

If the 'Z' is always present in the timestamp then just include it in the format string.

TIME_FORMAT = %Y-%m-%dT%H:%M:%S.%3NZ
---
If this reply helps you, an upvote would be appreciated.
0 Karma
Highlighted

Re: How to parse an unusual timestamp format?

New Member

Thanks, Rich. I tried that and several variants of the %3N bit at the end but they don't appear to have any effect -- my Splunk instance is ignoring it and ingesting events at "now". Do you have any idea why that might be? I am using a very high value for MAXTIMESTAMPLOOKAHEAD so I believe that is not the issue.

Are there any other props.conf config settings that might be interfering with this?

0 Karma
Highlighted

Re: How to parse an unusual timestamp format?

SplunkTrust
SplunkTrust

Ensure that you configure it on the parsing level, either HWF if available or IDX http://wiki.splunk.com/Where_do_I_configure_my_Splunk_settings and also make sure to restart the Splunk instance you configured.

Check using btool on the parsing instance to see if your props is applied correctly or if some other props.conf is taking precedence over it.

cheers, MuS

0 Karma
Highlighted

Re: How to parse an unusual timestamp format?

SplunkTrust
SplunkTrust

The Z is specific to Zulu time, i.e. UTC, i.e. GMT. In that same time zone specifier spot, you might see CDT for Central US Daylight Savings Time.

TIME_FORMAT = %Y-%m-%dT%H:%M:%S.%3N%Z

Please post the _raw timestamp from a couple of the events that did not parse, and we can analyze if there is another issue.

0 Karma