Anyone know if its possible to pull back the time from all the Splunk infrastructure?

[andynewsoncap]

Hello,

Anyone know if its possible to pull back the time from all the Splunk infrastructure.  I have over 200 IDX / SHD / DEP etc etc server.  In 4 Regions around the world.  And I think my NTP is failing / drifting.  And I what to show my IT Dept the problem if we have one.

So is it possible to ask, all the Splunk infrastructure the time.  So I can see / show at a glance oh that IDX server is 5 mins out from its cluster buddy's?

Thanks.

[PickleRick]

Typically you'd use an external infrastructure monitoring solution (like Nagios/Zabbix/whatever) to check the difference between host's time and a known reliable time source.

The idea of monitoring time is always tricky. Especially with splunk where everything is time related and you can't trust the server's time to... monitor the server's time 🙂

So you either have to check the server's reported time and verify it with an external reliable source (but you can't report it using simply timestamps on events since events have unknown delay; so you'd need another way to query them near real-time) or have the servers themselves query a reliable time source and report their findings along with their own timestamp. But tha boils down to querying a remote NTP which is what I understand is failing in your case.

[russellliss]

Hi @andynewsoncap

What about creating a custom app that executes:

Linux: date +%Y-%m-%d_%H:%M

Windows: Get-Date -format "yyyy-MM-dd_HH:mm"

and have your splunk components send the output to your indexers?

[gcusello]

Hi @andynewsoncap,

yes it's possible, but it isn't a configuration to do in Community, it requires the intervene of a Splunk Professional Service or al least a Splunk Architect with experience in multisite architectures.

with an infrastructure with your dimensions I hint to use Splunk PS!

ciao.

Giuseppe

[andynewsoncap]

I just smile at this point 🙂

[andynewsoncap]

Ok so what about asking is it possible to pull back the real time from one Splunk infrastructure component.

[gcusello]

Hi @andynewsoncap,

you could disable real time searches for your users, but it's a last solution.

I hint to try to set the correct timezone for the users, Splunk should manage different timezones.

I din't try with many Splunk servers, but with two Splunk Cloud Servers it correctly runs.

Ciao.

Giuseppe