Deployment Architecture

splunk 6.6.4 search head >> indexer peer down!!!


[ Environment ]
SPLUNK 6.6.4 Search Head Cluster 3 EA
SPLUNK 6.6.4 Indexer Cluster 4EA

Server specifications are very good.

[distsearch.conf] - searcher
connectionTimeout , sendTimeout , receiveTimeout
[limits.conf] - searcher , indexer
basemaxsearches ,maxrtsearchmultiplier , maxsearchespercpu

We have set this option to a value that is better than the default value.

If there are many searches on the search server, the indexer is not actually dead, but the error is still happening.

[error msg]
Unable to distribute to peer named * at uri=* using the uri-scheme=https because peer has status="Down". Please verify uri-scheme, connectivity to the search peer, that the search peer is up, and an adequate level of system resources are available. See the Troubleshooting Manual for more informati

[internal log]
ERROR StreamedSearch - sid=remote*adminadmin*_RMD561dff7a2ba7b014f1525229108.3657_03943F0E-D90E-4C81-BA9B-D2F89B841083, Broken pipe

WARN HttpListener - Socket error from ... while accessing /services/streams/search: Broken pipe

Do you know the cause of the symptom and the solution?

0 Karma


I'd guess that the indexer is under such heavy load that the connection times out.

Please show us your actual "better values" for the settings above, and also take a close look at your monitoring console about Ressource usage etc.

0 Karma


Thank you for your comment.

distsearch.conf has been doubled from its default value.

limits.conf The default has been increased by a factor of two or three, and the resources have not changed much.

However, the same symptoms.

0 Karma