Getting Data In

Incorrect event time in Splunk

brettcave
Builder

Hi, we recently changed platforms that we host some of our services on, and one of the changes included switching from a localized time on the server to UTC. Now, in Splunk, our times are all wrong.

I have a splunk forwarder installed on a host. If I check the date on servers, this is what I get:

splunkforwarder$ date
Mon Nov 11 13:37:35 UTC 2013

splunkindexer$ date
Mon Nov 11 08:37:45 EST 2013

The events being sent from the forwarder to the indexer look like this (sent 5 minutes before checking the server time):

Nov 11 13:29:57 127.0.0.1 Nov 11 13:29:57 ..... | ....

However, the Splunk indexer is marketing the time on the events as if the UTC time was the EST time, and rendering it locally (I am currently in a GMT+2 timezone, and my Splunk account is configured as such), so the events are marked as:

11/11/13 8:29:57.000 PM

which is what time it would be in my current time zone if it was 13:29 EST. Where is the time misinterpretation happening? On the source of the events (a java app), the splunk forwarder, or the splunk indexer?

Tags (2)
1 Solution

yannK
Splunk Employee
Splunk Employee

To investigate timezone issue, a good approach is use a search like this one
source=mysource host=myhost | convert ctime(_indextime) AS indextime | delay =_indextime-_time | table _time indextime date_zone host source sourcetype _raw

try alltime, and try realtime-alltime.

View solution in original post

brettcave
Builder

The times on the servers are right, but the indexer is parsing the UTC time on the forwarder as if it were EST. An event that occurred at 13h29m57s UTC is being reported by Splunk at 8:29:57PM GMT+2 (aka 6:29pm or 18h29 GMT) - it's 5 hours off.

If I change my user profile's time zone in Splunk to GMT, the lcoalized time is still off.

0 Karma

yannK
Splunk Employee
Splunk Employee

To investigate timezone issue, a good approach is use a search like this one
source=mysource host=myhost | convert ctime(_indextime) AS indextime | delay =_indextime-_time | table _time indextime date_zone host source sourcetype _raw

try alltime, and try realtime-alltime.

brettcave
Builder

Perfect, have now (finally) gotten around to testing this approach and it works great. I'd assume that this would only work if all sourcetypes (if set by sourcetype) would need to be on the same time zone for this to work effectively... is there not a method for determining the time by inspecting the time field and including the timezone in the timestamp of the event (log4j-based)

0 Karma

yannK
Splunk Employee
Splunk Employee

So the issue is pretty much the 5h difference between UTC (the forwarder) and EST (the indexer).

You have to enforce the timezone for this particular log in props.conf on the indexer. Using the more appropriate rule

  • by sourcetype
    [mysourcetype]
    TZ=UTC

  • by source
    [source::...mysource]
    TZ=UTC

  • by host
    [host:myforwarder]
    TZ=UTC

0 Karma

brettcave
Builder

Thanks for that yannK. Here's a sample of the data - all offsets are 18000.

_time indextime delay date_zone
11/13/13 5:54:06.000 PM 11/13/2013 12:54:04 -18002 local
11/13/13 5:38:44.000 PM 11/13/2013 12:38:42 -18002 local
11/13/13 5:32:55.000 PM 11/13/2013 12:32:54 -18001 local
11/13/13 5:32:54.000 PM 11/13/2013 12:33:04 -17990 local
11/13/13 5:32:54.000 PM 11/13/2013 12:32:54 -18000 local

0 Karma

somesoni2
SplunkTrust
SplunkTrust

Time on Forwarder (after switching to UTC):"Mon Nov 11 13:37:35 UTC 2013"
Time of Splunk Indexer:"Mon Nov 11 08:37:45 EST 2013"

Time on the logs (send from forwarder) sent at "Mon Nov 11 13:29:35 UTC 2013": Nov 11 13:29:57 [this is correct as its same time as time on forwarder.].
At this time, time on Splunk Indexer will be: "Mon Nov 11 08:37:45 EST 2013" which is same as what is shown as log's index time. I don't see any issue with interpretation.

Let me know if my understanding is incorrect.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...