topic Re: "ERROR DistributedPeerManagerHeartbeat - A time skew of approximately 61 seconds exists between this search head and peer indexer1" in Getting Data In

"ERROR DistributedPeerManagerHeartbeat - A time skew of approximately 61 seconds exists between this search head and peer indexer1"

sympatiko — Mon, 07 Sep 2015 02:58:52 GMT

Hi Splunkers,

I'm getting the following error on my search head's splunkd.log:

ERROR DistributedPeerManagerHeartbeat - A time skew of approximately 61 seconds exists between this search head and peer indexer1

I have a RF = 3 and SF = 3 setup.

Thanks

Re: "ERROR DistributedPeerManagerHeartbeat - A time skew of approximately 61 seconds exists between this search head and peer indexer1"

MuS — Mon, 07 Sep 2015 03:10:51 GMT

Hi sympatiko,

sound like some timing troubles, make sure your server are insync regarding time. See the docs http://docs.splunk.com/Documentation/Splunk/6.2.5/DistSearch/Searchheadpoolingconfigurationissues#Clock_skew_between_search_heads_and_shared_storage_can_affect_search_behavior for more details.

cheers, MuS

Re: "ERROR DistributedPeerManagerHeartbeat - A time skew of approximately 61 seconds exists between this search head and peer indexer1"

sympatiko — Mon, 07 Sep 2015 03:37:30 GMT

Thank you so much MuS.

Re: "ERROR DistributedPeerManagerHeartbeat - A time skew of approximately 61 seconds exists between this search head and peer indexer1"

dawid_schulz — Tue, 29 Sep 2020 07:34:55 GMT

Hello due to same problem I don't want to create new topic.

In log I found

ERROR DistributedPeerManagerHeartbeat - A time skew of approximately 1396 seconds exists between this search head and peer1
ERROR DistributedPeerManagerHeartbeat - A time skew of approximately 1909 seconds exists between this search head and peer2

But. When I check date I found that date is correct. 00:25:07 CEST on peer and 00:25:08 CEST on cluster master

I have All Data is Searchable Search Factor is Met Replication Factor is Not Met. Settings in server.conf are:

site_replication_factor = origin:1,site2:2,site4:2,total:4
site_search_factor = origin:1,site2:1,site4:1,total:2

We have 4 peers (showing under Indexer Clustering: Master Node) and 2 search (3 if we counting CM) and as you can see above - search factory is met (after a while) but replication factor is not. I also don't know what can happened if we have replication factor not met?

Ohh and one more importaint thing - this cause after we switch from single-site to multisite. Should I add in [clustering] replication_factor = 2 and search_factor = 1 to aviod replication factor not met ?

Thanks for any advice 😉

Re: "ERROR DistributedPeerManagerHeartbeat - A time skew of approximately 61 seconds exists between this search head and peer indexer1"

MuS — Thu, 15 Oct 2015 22:49:00 GMT

Your error states A time skew of approximately 1396 seconds exists between this search head and peer1 have you checked the time on the search heads and compared them to the indexers?

Re: "ERROR DistributedPeerManagerHeartbeat - A time skew of approximately 61 seconds exists between this search head and peer indexer1"

dawid_schulz — Thu, 15 Oct 2015 23:11:35 GMT

Wow - my mistake. I don't know how I can miss that. Sorry for bothering you !

Re: "ERROR DistributedPeerManagerHeartbeat - A time skew of approximately 61 seconds exists between this search head and peer indexer1"

MuS — Thu, 15 Oct 2015 23:15:58 GMT

That's why I have a fez 😛 Feel free to upvote my hints 😉

Re: "ERROR DistributedPeerManagerHeartbeat - A time skew of approximately 61 seconds exists between this search head and peer indexer1"

ppablo — Mon, 19 Oct 2015 04:19:12 GMT

*hint upvoted* Precisely why you have a fez ;D Cheers!

Re: "ERROR DistributedPeerManagerHeartbeat - A time skew of approximately 61 seconds exists between this search head and peer indexer1"

akhil4mdev — Tue, 19 Jun 2018 15:54:04 GMT

So how did you solve this ?

Re: "ERROR DistributedPeerManagerHeartbeat - A time skew of approximately 61 seconds exists between this search head and peer indexer1"

akhil4mdev — Tue, 19 Jun 2018 15:55:10 GMT

I am facing the same problem could u please help me , I am only getting "WARN" not an error and the skew is "76" seconds and I can search the data from the indexer though

Re: "ERROR DistributedPeerManagerHeartbeat - A time skew of approximately 61 seconds exists between this search head and peer indexer1"

MuS — Tue, 19 Jun 2018 20:12:54 GMT

Make sure all your Splunk instances are time synced using NTP or any similar service.

cheers, MuS