distributed system. splunk 7.1.2
one SH + one indexer
In the SH splunkd log:
DistributedPeerManager - Distributed: Unable to distribute to peer ..... using the uri-scheme=https because peer has status=2. Please verify uri-scheme, connectivity to the search peer, that the search peer is up, and an adequate level of system resources are available. See the Troubleshooting Manual for more information.
and it causes search failure.
what does status=2 mean? what might be happening here?
Any help is appreciated!
Just want to posted how we solved this issue in case other ppl see this issue as well - still on-going but less frequent
we suspect this is due to workload on indexer is too heavy. we dont have heavy forwarder in btw.
after we fixed couple of parsing issues on indexer. connection issue gets better.
You may want to revisit and ensure that right port used in your deployment.
Sometimes admin in config rush make mistake by sending logs to indexer on port 8089 instead of 9997 which is enough overwhelm the indexer.
status=2 is evaluated as "Unstable" and can only be triggered by the following two conditions.