How do you verify that multiple indexes are time synced?

[shandman]

We have upwards of 50 different security technologies reporting into Splunk. I'm being tasked with verifying that all the technologies reporting are properly time synced. Without going into each technology individually and verify NTP configurations, is there a way to run a query in Splunk to check time synchronization?

[pkeenan87]

You will need to look at how hosts are logging in your environment and how far off the timestamp of the last event sent by each host is from the current time. Here is a query that will help with that

| metadata type=hosts index=*
| eval time_zone_diff = Now() - lastTime
| eval recent_time = Now() - recentTime
| sort time_diff

Hosts with a negative value for the time_diff field are sending data that is "in the future" compared to the clock of your Splunk Servers. This indicates hosts with a time zone that are ahead of the time zone of your splunk infrastructure. Hosts with a high value for time_diff and a low value for recent_time indicate hosts with a time zone that are behind your Splunk infrastructure.

Here is the metadata doc for reference: https://docs.splunk.com/Documentation/Splunk/7.2.4/SearchReference/Metadata
Time zone docs for help with remediation: https://docs.splunk.com/Documentation/Splunk/7.2.4/Data/Applytimezoneoffsetstotimestamps