Incorrect time extraction

[ivan123357]

Hi!

I received an event with the following time string:  2023-12-12T13:39:25.400399Z CEF:0.....

This time is already in the correct timezone, but because of Z, splunk adds to 5 hours. I understand that Z it is timezone indicator but how i can ignore it?

Flow of this event is : Source --> HF --> Indexers. On HF or Indexers i dont have any props or transoforms settings. On Search Heads I extract a few fields from this event and it works. But i can't to extract this time correctly without Z.

I put the following regex inside props.conf on my SHs. Also i tried to put this on indexer's props.conf:

TIME_PREFIX = ^\d{2,4}-\d{1,2}-\d{1,2}T\d{1,2}:\d{1,2}:\d{1,2}\.\d{1,6}

 I tried to add TZ or TZ_ALIAS inside props.conf, but no effect. Where can I be wrong?

Thanks

[m_pham]

You need to add your index time configurations on the HF and not on the SH or Indexers since the HF is where your data is being parsed.

Your TIME_PREFIX configuration could be simpler but we would need to see a full sample log line to help with that; redact any sensitive information by the way if you provide a sample.

[ivan123357]

Yeah, sure

2023-12-12T19:39:25.400399Z <ip_address> CEF:0|vendor|product|version|AuditMessage_707|description|4|messageId=666 messageCategory=AuditMessage start=2023-12-12T19:39:25.400399Z user=user node=<ip_address> msg=<message>

[richgalloway]

Since the timestamp is at the beginning of the event, the prefix is a simple "^".  Try these settings

TIME_PREFIX = ^
TIME_FORMAT = %Y-%m-%dT%H:%M:%S.%6N%Z

The "%Z" is there to interpret the "Z" in the event as a time zone (UTC).

[m_pham]

Try this in your props.conf on your HF - I don't know the time zone of your log file so I assume it's UTC, change as needed:

TIME_PREFIX = ^
MAX_TIMESTAMP_LOOKAHEAD = 30
TIME_FORMAT = %FT%T.%6QZ
TZ = UTC

I highly recommend you read through best practices for parsing configurations here:

https://kinneygroup.com/blog/splunk-magic-8-props-conf/

Time variables can be found here: https://docs.splunk.com/Documentation/Splunk/latest/SearchReference/Commontimeformatvariables

[ivan123357]

Thanks for your answers. I tried both options.

I removed everything about time from SHs and Indexers. Put your config to HF (changed timezone),  restart splunkd process and generate a few new events. And again, it was indexed incorrect. I dont know why, but HF still thinking that time inside event is UTC, how i understood it is because timestamp inside event has "Z", but it is incorrect, this timestamp in my timezone, and splunk convert this correct timestamp into my timezone because he is thinking that it is UTC.

Timezone i used from this url : https://en.wikipedia.org/wiki/List_of_tz_database_time_zones

[richgalloway]

The presence of "Z" on the end of the timestamp means the time is in UTC.  If that is incorrect then the application writing the logs should be changed to use the correct time zone designation.

A workaround would be to remove "%Z" from the TIME_FORMAT setting so Splunk ignores the time zone.  It will default to the local (to the HF) time zone.