Deployment Architecture

Splunk Cluster Master Peer Handler Error

tprzelom
Path Finder

04-14-2014 13:03:09.199 -0700 ERROR ClusterMasterPeerHandler - Cannot add peer=x.x.x.x mgmtport=8089 (reason: non-zero pending job count=2090)

I've got walls of this error on the cluster master. Anyone know what's causing it and how serious of a problem it is?

Tags (3)

tprzelom
Path Finder

Initially the peer value consisted of all of my indexers, but as the count value wound down I was left with 2 peers showing the errors with a count value of less than 10.

Those 2 peers showed the following in their logs repeatedly.

04-17-2014 11:02:03.853 -0700 WARN CMSlave - handleHeartbeatDone: successful heartbeat and re-add not received but proxy is in disconnected state. Forcing re-add.
04-17-2014 11:02:03.853 -0700 INFO CMSlave - event=addPeer resetting masks for all buckets on clearAndReadd

Eventually the error subsided which I attribute to the cluster reaching a state of homeostasis with its replication rate because my search and replication factors are not met yet.

Got questions? Get answers!

Join the Splunk Community Slack to learn, troubleshoot, and make connections with fellow Splunk practitioners in real time!

Meet up IRL or virtually!

Join Splunk User Groups to connect and learn in-person by region or remotely by topic or industry.

Get Updates on the Splunk Community!

Announcing Modern Navigation: A New Era of Splunk User Experience

We are excited to introduce the Modern Navigation feature in the Splunk Platform, available to both cloud and ...

Modernize your Splunk Apps – Introducing Python 3.13 in Splunk

We are excited to announce that the upcoming releases of Splunk Enterprise 10.2.x and Splunk Cloud Platform ...

Step into “Hunt the Insider: An Splunk ES Premier Mystery” to catch a cybercriminal ...

After a whole week of being on call, you fell asleep on your keyboard, and you hit a sequence of buttons that ...