Getting Data In

When trying to set up a distributed system, can you help me with the following error?: "Unable to distribute to peer, peer has status=2"

xindeNokia
Path Finder

distributed system. splunk 7.1.2
one SH + one indexer

In the SH splunkd log:

DistributedPeerManager - Distributed: Unable to distribute to peer ..... using the uri-scheme=https because peer has status=2. Please verify uri-scheme, connectivity to the search peer, that the search peer is up, and an adequate level of system resources are available. See the Troubleshooting Manual for more information.

and it causes search failure.

what does status=2 mean? what might be happening here?

Any help is appreciated!

bgronvall_splun
Splunk Employee
Splunk Employee

status=2 is evaluated as "Unstable" and can only be triggered by the following two conditions.

  1. There is a time skew between the SH and Search Peer.
  2. The indexer is oversubscribed and rate at which it returns results is inconsistent with the other search peers.
0 Karma

xindeNokia
Path Finder

Just want to posted how we solved this issue in case other ppl see this issue as well - still on-going but less frequent

we suspect this is due to workload on indexer is too heavy. we dont have heavy forwarder in btw.
after we fixed couple of parsing issues on indexer. connection issue gets better.

0 Karma

woodcock
Esteemed Legend

Please do click Accept on your answer.

0 Karma

cybermonday
Explorer

You may want to revisit and ensure that right port used in your deployment.

Sometimes admin in config rush make mistake by sending logs to indexer on port 8089 instead of 9997 which is enough overwhelm the indexer.

0 Karma
.conf21 Now Fully Virtual!
Register for FREE Today!

We've made .conf21 totally virtual and totally FREE! Our completely online experience will run from 10/19 through 10/20 with some additional events, too!