Deployment Architecture

Splunk Cluster Master Peer Handler Error

tprzelom
Path Finder

04-14-2014 13:03:09.199 -0700 ERROR ClusterMasterPeerHandler - Cannot add peer=x.x.x.x mgmtport=8089 (reason: non-zero pending job count=2090)

I've got walls of this error on the cluster master. Anyone know what's causing it and how serious of a problem it is?

Tags (3)

tprzelom
Path Finder

Initially the peer value consisted of all of my indexers, but as the count value wound down I was left with 2 peers showing the errors with a count value of less than 10.

Those 2 peers showed the following in their logs repeatedly.

04-17-2014 11:02:03.853 -0700 WARN CMSlave - handleHeartbeatDone: successful heartbeat and re-add not received but proxy is in disconnected state. Forcing re-add.
04-17-2014 11:02:03.853 -0700 INFO CMSlave - event=addPeer resetting masks for all buckets on clearAndReadd

Eventually the error subsided which I attribute to the cluster reaching a state of homeostasis with its replication rate because my search and replication factors are not met yet.

Got questions? Get answers!

Join the Splunk Community Slack to learn, troubleshoot, and make connections with fellow Splunk practitioners in real time!

Meet up IRL or virtually!

Join Splunk User Groups to connect and learn in-person by region or remotely by topic or industry.

Get Updates on the Splunk Community!

Your Feedback. Our Roadmap. Visit the PX Feedback Booth at .conf26

You use Splunk every day, come and help shape what's next.  Save Your Seat: Product-Focused Sessions at ...

Agentic SOC Triage: Investigating Splunk ES Notables with MCP Server and a Local LLM

The Problem: Too Many Alerts, Too Little Context Security operations teams running Splunk Enterprise Security ...

Painting a Clearer Picture: Creating Cross-Domain Visibility with AI Canvas

Watch Now Painting a Clearer Picture: Creating Cross-Domain Visibility with AI Canvas     Do you ever feel ...