Getting Data In

"ERROR DistributedPeerManagerHeartbeat - A time skew of approximately 61 seconds exists between this search head and peer indexer1"

Communicator

Hi Splunkers,

I'm getting the following error on my search head's splunkd.log:

ERROR DistributedPeerManagerHeartbeat - A time skew of approximately 61 seconds exists between this search head and peer indexer1

I have a RF = 3 and SF = 3 setup.

Thanks

0 Karma
1 Solution

SplunkTrust
SplunkTrust

Hi sympatiko,

sound like some timing troubles, make sure your server are insync regarding time. See the docs http://docs.splunk.com/Documentation/Splunk/6.2.5/DistSearch/Searchheadpoolingconfigurationissues#Cl... for more details.

cheers, MuS

View solution in original post

SplunkTrust
SplunkTrust

Hi sympatiko,

sound like some timing troubles, make sure your server are insync regarding time. See the docs http://docs.splunk.com/Documentation/Splunk/6.2.5/DistSearch/Searchheadpoolingconfigurationissues#Cl... for more details.

cheers, MuS

View solution in original post

Explorer

Hello due to same problem I don't want to create new topic.

In log I found

ERROR DistributedPeerManagerHeartbeat - A time skew of approximately 1396 seconds exists between this search head and peer1
ERROR DistributedPeerManagerHeartbeat - A time skew of approximately 1909 seconds exists between this search head and peer2

But. When I check date I found that date is correct. 00:25:07 CEST on peer and 00:25:08 CEST on cluster master

I have All Data is Searchable Search Factor is Met Replication Factor is Not Met. Settings in server.conf are:

sitereplicationfactor = origin:1,site2:2,site4:2,total:4
sitesearchfactor = origin:1,site2:1,site4:1,total:2

We have 4 peers (showing under Indexer Clustering: Master Node) and 2 search (3 if we counting CM) and as you can see above - search factory is met (after a while) but replication factor is not. I also don't know what can happened if we have replication factor not met?

Ohh and one more importaint thing - this cause after we switch from single-site to multisite. Should I add in [clustering] replicationfactor = 2 and searchfactor = 1 to aviod replication factor not met ?

Thanks for any advice 😉

0 Karma

SplunkTrust
SplunkTrust

Your error states A time skew of approximately 1396 seconds exists between this search head and peer1 have you checked the time on the search heads and compared them to the indexers?

Explorer

Wow - my mistake. I don't know how I can miss that. Sorry for bothering you !

0 Karma

SplunkTrust
SplunkTrust

That's why I have a fez 😛 Feel free to upvote my hints 😉

Community Manager
Community Manager

*hint upvoted* Precisely why you have a fez ;D Cheers!

0 Karma

Explorer

So how did you solve this ?

0 Karma

Explorer

I am facing the same problem could u please help me , I am only getting "WARN" not an error and the skew is "76" seconds and I can search the data from the indexer though

0 Karma

SplunkTrust
SplunkTrust

Make sure all your Splunk instances are time synced using NTP or any similar service.

cheers, MuS

0 Karma

Communicator

Thank you so much MuS.

0 Karma