Error message when running a search on the search head - Unable to distribute to peer

[kiril123]

I get the following error message when running a search on the search head:

Unable to distribute to peer named :8089 at uri=:8089 using the uri-scheme=https because peer has status="Down". Please verify uri-scheme, connectivity to the search peer, that the search peer is up, and an adequate level of system resources are available. See the Troubleshooting Manual for more information.

I've tried increasing timeout settings in distsearch.conf with no luck.

I have also checked the system resources on the search head and the indexers and didn't see any constraint.

Do you have any ideas?

[hardikJsheth · Accepted solution]

The reason you get this error is because, search head is unable to distribute searches to indexer nodes. As explained by @Sivamedis, this error comes when your indexer are too busy / highly congested.

To resolve this error, can you check if any real time scheduled searches / scheduled searches which are running for quite long period of time and try to fine tune these searches. To find searches which are taking most time you can use following query:

index=_audit action=search search_id=* | regex total_run_time="[0-9\.]+" | eval total_run_time=coalesce(total_run_time,"") | stats sum(total_run_time) as total_run_time, avg(total_run_time) as avg_run_time by savedsearch_name, user | sort -total_run_time

[Sivamedis]

I did a research on this, came to know that
Indexers are not actually down but periodically get busy enough that the response is slowed so that they "appear" down for brief periods of time. This needs some configuration changes and those changes will depend on diag and ptrace.

[xisura]

It seems that your searchhead cannot established connection to the search peer .Did you check the connectivity ? , is the search peer reachable?
check also if the port 8089 is blocked or not.
check the firewall or in linux iptables -f

[kiril123]

I forgot to add that the issue is intermittent. I can run the searches, however at a time of high load search head loses connectivity to the indexers.

[andre_tucker]

It looks like you are setup to use distributed search but the indexer's address was never provided. If you go to settings >
"distributed search" then you should be able to see what indexers your SH is configured to search from. If you do not have one set then you can click add new and then input the uri scheme for the indexer (https://indexer_hostname_or_ip:8089) this is assuming the default management port has not been changed on your system. Then you will need to supply a splunk admin password. If the admin account is still set to "changeme" on your system then this will not work so you will need to change it (as suggested).

See the link below for documentation.
http://docs.splunk.com/Documentation/Splunk/6.6.3/DistSearch/Configuredistributedsearch