Deployment Architecture

splunk 6.6.4 search head >> indexer peer down!!!

Explorer

[ Environment ]
SPLUNK 6.6.4 Search Head Cluster 3 EA
SPLUNK 6.6.4 Indexer Cluster 4EA

Server specifications are very good.

[distsearch.conf] - searcher
connectionTimeout , sendTimeout , receiveTimeout
[limits.conf] - searcher , indexer
basemaxsearches ,maxrtsearchmultiplier , maxsearchespercpu

We have set this option to a value that is better than the default value.

If there are many searches on the search server, the indexer is not actually dead, but the error is still happening.

[error msg]
Unable to distribute to peer named * at uri=* using the uri-scheme=https because peer has status="Down". Please verify uri-scheme, connectivity to the search peer, that the search peer is up, and an adequate level of system resources are available. See the Troubleshooting Manual for more informati

[internal log]
ERROR StreamedSearch - sid=remote*adminadmin*_RMD561dff7a2ba7b014f1525229108.3657_03943F0E-D90E-4C81-BA9B-D2F89B841083, Broken pipe

WARN HttpListener - Socket error from ... while accessing /services/streams/search: Broken pipe

Do you know the cause of the symptom and the solution?

0 Karma

SplunkTrust
SplunkTrust

I'd guess that the indexer is under such heavy load that the connection times out.

Please show us your actual "better values" for the settings above, and also take a close look at your monitoring console about Ressource usage etc.

0 Karma

Explorer

Thank you for your comment.

distsearch.conf has been doubled from its default value.

limits.conf The default has been increased by a factor of two or three, and the resources have not changed much.

However, the same symptoms.

0 Karma