Getting Data In

How do I determine if hosts are in the wrong timezone?

Jason
Motivator

I have finished bringing in any archive data that I need to. Now all data flowing into Splunk is be up-to-the-minute, whether it be from a forwarder, a network port, or a script.

How do I tell if certain hosts sending data to Splunk have incorrect timezones?

Tags (3)
0 Karma
1 Solution

Jason
Motivator

This is a very important question, as normalization of time is essential so you can have an accurate picture of what actually happened at a certain point in time.

(If you have a datasource in Central time, but your indexer is in Eastern and wasn't told otherwise, it will index the data as Eastern and events will show in Splunk as an hour behind other events that happened that minute!)

One method is to try running this search. Run it over a period that would capture at least one event from each host, such as a day. It compares _time (the epoch time version of the time extracted from the event, adjusted with any timezone in the event or in props.conf) to _indextime (the epoch time version of when Splunk indexed the event). If your data is all current, these should be only seconds off if a datasource is properly configured.

* | dedup host | eval timediff=(_time-_indextime) | eval HoursOff=round(timediff/3600) | search NOT HoursOff=0 | rename _indextime as IndexTime | eval Time=_time | table Time, IndexTime, HoursOff, host

Remember this checks Index Time against Time extracted from the event. It assumes the most recent event from each host is coming in at current time, and isn't archive data.

I'm going to get this into an App, along with a script to add the timezone into the source field for troublesome linux servers. At some point.

View solution in original post

Jason
Motivator

This is a very important question, as normalization of time is essential so you can have an accurate picture of what actually happened at a certain point in time.

(If you have a datasource in Central time, but your indexer is in Eastern and wasn't told otherwise, it will index the data as Eastern and events will show in Splunk as an hour behind other events that happened that minute!)

One method is to try running this search. Run it over a period that would capture at least one event from each host, such as a day. It compares _time (the epoch time version of the time extracted from the event, adjusted with any timezone in the event or in props.conf) to _indextime (the epoch time version of when Splunk indexed the event). If your data is all current, these should be only seconds off if a datasource is properly configured.

* | dedup host | eval timediff=(_time-_indextime) | eval HoursOff=round(timediff/3600) | search NOT HoursOff=0 | rename _indextime as IndexTime | eval Time=_time | table Time, IndexTime, HoursOff, host

Remember this checks Index Time against Time extracted from the event. It assumes the most recent event from each host is coming in at current time, and isn't archive data.

I'm going to get this into an App, along with a script to add the timezone into the source field for troublesome linux servers. At some point.

Get Updates on the Splunk Community!

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...