Need help with time synchronization , time stamp events by the hosts. Reporting event in the past or future. Thank a lot

[SamHTexas]

I use the below SPL to find how hosts are logging in my environment and how far off the timestamp of the last event sent by each host is from the current time.

| eval time_zone_diff = Now() - lastTime

| eval recent_time = Now() - recentTime

| sort time_diff

The results are hard to read. Instead of reporting which hosts, it just shows so many are late & so many are ahead & no host is listed. Would you help with an SPL, not Meta Woot! App (could not configure it properly) to list which hosts are late & which are reporting event in the future please. Thanks a million in advance.

[PickleRick]

It's not that easy, unfortunately. One thing is that you can find the difference between _time and _indextime For example, by doing:

| tstats latest(_time) as etime latest(_indextime) as itime where index=* by source index host
| eval timediff=itime-etime 
| fieldformat timediff=tostring(timediff,"duration")
| table host index source timediff

But it will show you only how much time passed between the time "in" the event (which might be parsed out of the event, supplied by forwarder, or even explicitly set by source in case of - for example - HEC inputs) and the time at which the event was indexed. It's not a general indication of a time drift between your infrastructure components.

There might be, for example, some sources for which you get data in batches (like reading whole logfiles supplied daily for the whole "yesterday" containing events from throughout the day or WEC subscriptions which update Forwarded Logs at predefined intervals).

So the subject of keeping your time synchronized across your whole infrastructure is not up to splunk and is not really monitorable by splunk as such. It's the other way around - the splunk infrastructure relies on time being set up consistently across all your devices.

Comparing _time and _indextime can however help with finding sources which have problems with timezone definitions.