topic Re: Incorrect event time in Splunk in Getting Data In

Incorrect event time in Splunk

brettcave — Mon, 11 Nov 2013 13:46:45 GMT

Hi, we recently changed platforms that we host some of our services on, and one of the changes included switching from a localized time on the server to UTC. Now, in Splunk, our times are all wrong.

I have a splunk forwarder installed on a host. If I check the date on servers, this is what I get:

splunkforwarder$ date
Mon Nov 11 13:37:35 UTC 2013

splunkindexer$ date
Mon Nov 11 08:37:45 EST 2013

The events being sent from the forwarder to the indexer look like this (sent 5 minutes before checking the server time):

Nov 11 13:29:57 127.0.0.1 Nov 11 13:29:57 ..... | ....

However, the Splunk indexer is marketing the time on the events as if the UTC time was the EST time, and rendering it locally (I am currently in a GMT+2 timezone, and my Splunk account is configured as such), so the events are marked as:

11/11/13 8:29:57.000 PM

which is what time it would be in my current time zone if it was 13:29 EST. Where is the time misinterpretation happening? On the source of the events (a java app), the splunk forwarder, or the splunk indexer?

Re: Incorrect event time in Splunk

somesoni2 — Mon, 11 Nov 2013 16:37:37 GMT

Time on Forwarder (after switching to UTC):"Mon Nov 11 13:37:35 UTC 2013"
Time of Splunk Indexer:"Mon Nov 11 08:37:45 EST 2013"

Time on the logs (send from forwarder) sent at "Mon Nov 11 13:29:35 UTC 2013": Nov 11 13:29:57 [this is correct as its same time as time on forwarder.].
At this time, time on Splunk Indexer will be: "Mon Nov 11 08:37:45 EST 2013" which is same as what is shown as log's index time. I don't see any issue with interpretation.

Let me know if my understanding is incorrect.

Re: Incorrect event time in Splunk

yannK — Mon, 11 Nov 2013 17:45:18 GMT

To investigate timezone issue, a good approach is use a search like this one
source=mysource host=myhost | convert ctime(_indextime) AS indextime | delay =_indextime-_time | table _time indextime date_zone host source sourcetype _raw

try alltime, and try realtime-alltime.

Re: Incorrect event time in Splunk

brettcave — Mon, 28 Sep 2020 15:16:05 GMT

Thanks for that yannK. Here's a sample of the data - all offsets are 18000.

_time indextime delay date_zone
11/13/13 5:54:06.000 PM 11/13/2013 12:54:04 -18002 local
11/13/13 5:38:44.000 PM 11/13/2013 12:38:42 -18002 local
11/13/13 5:32:55.000 PM 11/13/2013 12:32:54 -18001 local
11/13/13 5:32:54.000 PM 11/13/2013 12:33:04 -17990 local
11/13/13 5:32:54.000 PM 11/13/2013 12:32:54 -18000 local

Re: Incorrect event time in Splunk

brettcave — Wed, 13 Nov 2013 11:13:47 GMT

The times on the servers are right, but the indexer is parsing the UTC time on the forwarder as if it were EST. An event that occurred at 13h29m57s UTC is being reported by Splunk at 8:29:57PM GMT+2 (aka 6:29pm or 18h29 GMT) - it's 5 hours off.

If I change my user profile's time zone in Splunk to GMT, the lcoalized time is still off.

Re: Incorrect event time in Splunk

yannK — Wed, 13 Nov 2013 16:28:50 GMT

So the issue is pretty much the 5h difference between UTC (the forwarder) and EST (the indexer).

You have to enforce the timezone for this particular log in props.conf on the indexer. Using the more appropriate rule

by sourcetype
[mysourcetype]
TZ=UTC
by source
[source::...mysource]
TZ=UTC
by host
[host:myforwarder]
TZ=UTC

Re: Incorrect event time in Splunk

brettcave — Thu, 02 Jan 2014 14:42:06 GMT

Perfect, have now (finally) gotten around to testing this approach and it works great. I'd assume that this would only work if all sourcetypes (if set by sourcetype) would need to be on the same time zone for this to work effectively... is there not a method for determining the time by inspecting the time field and including the timezone in the timestamp of the event (log4j-based)