"ERROR DistributedPeerManagerHeartbeat - A time skew of approximately 61 seconds exists between this search head and peer indexer1"

[sympatiko]

Hi Splunkers,

I'm getting the following error on my search head's splunkd.log:

ERROR DistributedPeerManagerHeartbeat - A time skew of approximately 61 seconds exists between this search head and peer indexer1

I have a RF = 3 and SF = 3 setup.

Thanks

[MuS · Accepted solution]

Hi sympatiko,

sound like some timing troubles, make sure your server are insync regarding time. See the docs http://docs.splunk.com/Documentation/Splunk/6.2.5/DistSearch/Searchheadpoolingconfigurationissues#Cl... for more details.

cheers, MuS

[dawid_schulz]

Hello due to same problem I don't want to create new topic.

In log I found

ERROR DistributedPeerManagerHeartbeat - A time skew of approximately 1396 seconds exists between this search head and peer1
ERROR DistributedPeerManagerHeartbeat - A time skew of approximately 1909 seconds exists between this search head and peer2

But. When I check date I found that date is correct. 00:25:07 CEST on peer and 00:25:08 CEST on cluster master

I have All Data is Searchable Search Factor is Met Replication Factor is Not Met. Settings in server.conf are:

site_replication_factor = origin:1,site2:2,site4:2,total:4
site_search_factor = origin:1,site2:1,site4:1,total:2

We have 4 peers (showing under Indexer Clustering: Master Node) and 2 search (3 if we counting CM) and as you can see above - search factory is met (after a while) but replication factor is not. I also don't know what can happened if we have replication factor not met?

Ohh and one more importaint thing - this cause after we switch from single-site to multisite. Should I add in [clustering] replication_factor = 2 and search_factor = 1 to aviod replication factor not met ?

Thanks for any advice 😉

[MuS]

Your error states A time skew of approximately 1396 seconds exists between this search head and peer1 have you checked the time on the search heads and compared them to the indexers?

[dawid_schulz]

Wow - my mistake. I don't know how I can miss that. Sorry for bothering you !

[MuS]

That's why I have a fez 😛 Feel free to upvote my hints 😉

[ppablo]

*hint upvoted* Precisely why you have a fez ;D Cheers!

[akhil4mdev]

So how did you solve this ?

[akhil4mdev]

I am facing the same problem could u please help me , I am only getting "WARN" not an error and the skew is "76" seconds and I can search the data from the indexer though

[MuS]

Make sure all your Splunk instances are time synced using NTP or any similar service.

cheers, MuS

[sympatiko]

Thank you so much MuS.