Deployment Architecture

Error message when running a search on the search head - Unable to distribute to peer

kiril123
Path Finder

I get the following error message when running a search on the search head:

Unable to distribute to peer named :8089 at uri=:8089 using the uri-scheme=https because peer has status="Down". Please verify uri-scheme, connectivity to the search peer, that the search peer is up, and an adequate level of system resources are available. See the Troubleshooting Manual for more information.

I've tried increasing timeout settings in distsearch.conf with no luck.

I have also checked the system resources on the search head and the indexers and didn't see any constraint.

Do you have any ideas?

0 Karma
1 Solution

hardikJsheth
Motivator

The reason you get this error is because, search head is unable to distribute searches to indexer nodes. As explained by @Sivamedis, this error comes when your indexer are too busy / highly congested.

To resolve this error, can you check if any real time scheduled searches / scheduled searches which are running for quite long period of time and try to fine tune these searches. To find searches which are taking most time you can use following query:

index=_audit action=search search_id=* | regex total_run_time="[0-9\.]+" | eval total_run_time=coalesce(total_run_time,"") | stats sum(total_run_time) as total_run_time, avg(total_run_time) as avg_run_time by savedsearch_name, user | sort -total_run_time

View solution in original post

0 Karma

hardikJsheth
Motivator

The reason you get this error is because, search head is unable to distribute searches to indexer nodes. As explained by @Sivamedis, this error comes when your indexer are too busy / highly congested.

To resolve this error, can you check if any real time scheduled searches / scheduled searches which are running for quite long period of time and try to fine tune these searches. To find searches which are taking most time you can use following query:

index=_audit action=search search_id=* | regex total_run_time="[0-9\.]+" | eval total_run_time=coalesce(total_run_time,"") | stats sum(total_run_time) as total_run_time, avg(total_run_time) as avg_run_time by savedsearch_name, user | sort -total_run_time
0 Karma

Sivamedis
New Member

I did a research on this, came to know that
Indexers are not actually down but periodically get busy enough that the response is slowed so that they "appear" down for brief periods of time. This needs some configuration changes and those changes will depend on diag and ptrace.

0 Karma

xisura
Communicator

It seems that your searchhead cannot established connection to the search peer .Did you check the connectivity ? , is the search peer reachable?
check also if the port 8089 is blocked or not.
check the firewall or in linux iptables -f

0 Karma

kiril123
Path Finder

I forgot to add that the issue is intermittent. I can run the searches, however at a time of high load search head loses connectivity to the indexers.

0 Karma

andre_tucker
Path Finder

It looks like you are setup to use distributed search but the indexer's address was never provided. If you go to settings >
"distributed search" then you should be able to see what indexers your SH is configured to search from. If you do not have one set then you can click add new and then input the uri scheme for the indexer (https://indexer_hostname_or_ip:8089) this is assuming the default management port has not been changed on your system. Then you will need to supply a splunk admin password. If the admin account is still set to "changeme" on your system then this will not work so you will need to change it (as suggested).

See the link below for documentation.
http://docs.splunk.com/Documentation/Splunk/6.6.3/DistSearch/Configuredistributedsearch

0 Karma
Get Updates on the Splunk Community!

Enterprise Security Content Update (ESCU) | New Releases

In November, the Splunk Threat Research Team had one release of new security content via the Enterprise ...

Index This | Divide 100 by half. What do you get?

November 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with this ...

Stay Connected: Your Guide to December Tech Talks, Office Hours, and Webinars!

❄️ Celebrate the season with our December lineup of Community Office Hours, Tech Talks, and Webinars! ...