Where to check the clock skew?

[vrmandadi]

I have issues with clustering .I tried running the resync command but it says "Downloaded an old snapshot created 10018 seconds ago; Check for clock skew on this member or the captain; If no clock skew is found, check the captain for possible snapshot creation failures
"

Where can I see the clock skew and where can I change it

[amitm05]

Hi @vrmandadi

You can run this to check for time skew among your instances -

| rest /services/server/info | eval updated_t=round(strptime(updated, "%Y-%m-%dT%H:%M:%S%z"), 2) | eval delta=now()-updated_t | table serverName, updated, delta | convert ctime(updated_t)| rename updated as "Local Time on Server on Request" delta as "Offset - Time in Seconds"

And if you find some gap here, you might want to set your server clocks
Also check -
https://docs.splunk.com/Documentation/Splunk/7.3.0/DistSearch/Troubleshootdistributedsearch

[fredclown]

I'm not sure this works any more. The update field that is returned from the REST API is always set to"1969-12-31T16:00:00-08:00" for all my servers. Maybe it's just my environment. Anyway, here is an alternate solution.

| rest splunk_server=* /services/search/jobs/export search="| makeresults count=1 | rename _time as time" output_mode=csv
| makemv tokenizer="([^\n]+)" value
| eval local_time=now()
| eval remote_time=mvindex(value,1)
| eval local_time_formatted=strftime(local_time, "%Y-%m-%d %H:%M:%S")
| eval remote_time_formatted=strftime(remote_time, "%Y-%m-%d %H:%M:%S")
| eval delta_secs=abs(local_time-remote_time)
| fields splunk_server, local_time_formatted, remote_time_formatted, delta_secs
| sort - delta_secs

[PickleRick]

To be honest, I know that if you have Splunk you probably want to do everything in Splunk but monitoring the time sync state is something I'd rather do at OS level using your favourite monitoring software (zabbix, nagios, whatever).

[amitm05]

Let me know by accepting the answer if it responds your query. Thanks

[lmvmandadi]

I checked the time but the offset-time in seconds did not show anything .But still the error persists