Why am I getting the error"Unable to distribute to peer named X.X.X.X:YYYY at uri=X.X.X.X:YYYY using the uri-scheme=https because peer has status="Down" on my search head for all queries?

[pavanae]

I am getting the below error on my search head for all the queries.

"Unable to distribute to peer named X.X.X.X:PPPP  at uri=X.X.X.X:PPPP using the uri-scheme=https because peer has status="Down". Please verify uri-scheme, connectivity to the search peer, that the search peer is up, and an adequate level of system resources are available. See the Troubleshooting Manual for more information."

Where X.X.X.X is indexer IP address and PPPP is port.

When I tried to look what's happening in the internal logs I only see the below warning message on all the events

Query used :-index="_internal" source=/cs/splunk/search/var/log/splunk/splunkd.log X.X.X.X

Warning log :- 02-24-2018 16:35:15.331 +0100 WARN  DistributedPeerManager - Unable to distribute to peer named X.X.X.X:PPPP at uri=X.X.X.X:PPPP using the uri-scheme=https because peer has status="Down"

Please verify uri-scheme, connectivity to the search peer, that the search peer is up, and an adequate level of system resources are available. See the Troubleshooting Manual for more information.

I have tried the below to identify the issue

1. removing the cluster and re-adding it doesn’t resolve this problem
2. Rebooting the indexer doesn't help
3. I can see some buckets showing up on indexer when searching it, however, most of them are unavailable.

Could anyone suggest what exactly could be the issue?

Note :- search head version :- 6.5.3 and indexer version for host X.X.X.X is : 6.6.5

I know that upgrading the search head to the version 6.6.5 would resolve this but I'm trying to find the reason why it fails when the search head version is lesser than the indexer version

[DBattisto]

I know this has been marked as solved, but if anyone else finds themselves here...I have recently rebuilt a Splunk searchhead VM and decided to copy over the $splunk/etc/ file to the new VM to minimize downtime. I was receiving this error and had to go to 'settings>distributed search' and reauthenticate to the indexer.

[JDukeSplunk · Accepted solution]

My first thought would be SSL negotiation. You might want to check what sslVersions your authentication.conf and server.conf have on the boxes. If the system marking the other "down" only accepts ssl3 and above, but the "down" server is transmitting using ssl2 or ssl1 the receiving server will not be able to negotiate a comm channel with it, and it will fail.

[anil1432]

Hello @JDukeSplunk  ,

Let Me Know What Is The Process  If It fails? , We Might Be  able to Do Any Settings,

[adonio]

hello there,

the question "why it fails?" is probably to product builders / engineers.
would recommend to do what you know needs to be done and match your versions of splunk components.
some answers and articles in the following links:
https://answers.splunk.com/answers/484456/using-different-splunk-version-for-search-head-clu.htmlhtt...
http://docs.splunk.com/Documentation/Splunk/6.5.1/Indexer/Systemrequirements#Compatibility_between_p...

hope it helps