When trying to set up a distributed system, can you help me with the following error?: "Unable to distribute to peer, peer has status=2"

[xindeNokia]

distributed system. splunk 7.1.2
one SH + one indexer

In the SH splunkd log:

DistributedPeerManager - Distributed: Unable to distribute to peer ..... using the uri-scheme=https because peer has status=2. Please verify uri-scheme, connectivity to the search peer, that the search peer is up, and an adequate level of system resources are available. See the Troubleshooting Manual for more information.

and it causes search failure.

what does status=2 mean? what might be happening here?

Any help is appreciated!

[bgronvall_splun]

status=2 is evaluated as "Unstable" and can only be triggered by the following two conditions.

1. There is a time skew between the SH and Search Peer.
2. The indexer is oversubscribed and rate at which it returns results is inconsistent with the other search peers.

[xindeNokia]

Just want to posted how we solved this issue in case other ppl see this issue as well - still on-going but less frequent

we suspect this is due to workload on indexer is too heavy. we dont have heavy forwarder in btw.
after we fixed couple of parsing issues on indexer. connection issue gets better.

[woodcock]

Please do click Accept on your answer.

[cybermonday]

You may want to revisit and ensure that right port used in your deployment.

Sometimes admin in config rush make mistake by sending logs to indexer on port 8089 instead of 9997 which is enough overwhelm the indexer.