System clock not identical in SH and indexer.

[arun_kant_sharm]

Hi Experts,

I am getting below error in Setting -> Search peers ->Distributed search in my Splunk Search heads and Splunk deployment server.

In Deployment Server:
The times on the system clocks for the machines running this search head and the intended search peer at uri=https://:8089 differ from each other.
You must make the times identical, so that time comparisons are valid across the deployment. skew_seconds=250.
The max tolerable skew may be configured in [search]:max_tolerable_skew in limits.conf on the search head. Last Connect Time:2019-09-11T06:36:57.000+00:00; Failed 11 out of 11 times.

In Search heads:
Error [00000040] Instance name "" The times on the system clocks for the machines running this search head and the intended search peer at uri=https://:8089 differ from each other.
You must make the times identical, so that time comparisons are valid across the deployment. skew_seconds=132. The max tolerable skew may be configured in [search]:max_tolerable_skew in limits.conf on the search head.
Last Connect Time:2019-09-11T06:45:08.000+00:00; Failed 8 out of 8 times.

I already updated limits.conf file on the Search Head (path: /opt/splunk/etc/system/local).

[search]
remote_timeline_fetchall = 0

Please suggest what i need to do to fix it?

[arun_kant_sharm · Accepted solution]

I configured chrony insted of NTP, actually my all SH, indexers and deployment server are on AWS EC2 instance.
I already configured ntp.conf to configure NTP, but unfortunately its not work in my case.
Then I configured crony in all my EC2 instances, and its work properly now.
https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/set-time.html

[chinmoya]

Check the below link:

https://answers.splunk.com/answers/755087/search-peers-sick.html#answer-759155

[harsmarvania57]

Hi,

You need to discuss with your OS Administrator to fix time skew issue on OS, it is mostly due to NTP is not configured on your server or it is not working as expected.

I am not sure why you set remote_timeline_fetchall = 0 in limits.conf, this setting doesn't have any relation with clock skew errors in your splunk instances.