Why am I getting the following "send failure" message in my internal logs: "pushing PK to search peer" ?

[jelizabeth]

Here is the complete warning message:

Send failure while pushing PK to search peer = https://*.*.*.*:8089 , Read Timeout

I'm getting the above warning messages in the internal Splunk logs every minute from each of our 3 search heads.

The search peer in question is in our secondary site (let's say B) to the search heads (site A), but there are two other search peers in the same site (B) which we don't get any warning messages for.

I've done a ping and netcat from each of the search heads in site A to each of the three search peers in Site B and the results are the same for each one, connection established and similar ping times.

It's not a connection issue, so i'm wondering what else could be causing it?

[harsmarvania57]

It looks like you have some network issues between site A and site B (Maybe high latency). Same problem faced by other user previously and for them it was network issue. (reference : https://answers.splunk.com/answers/455635/why-is-my-search-head-cluster-not-working-after-up-1.html)

[jelizabeth]

As previously stated we don't believe it's a network issue as all tests between instances show no latency. we are looking for an alternative reason as to what could be causing the issue.

[harsmarvania57]

In that case you can directly distribute key files using process given here in Splunk Docs and after that check again whether splunk on Search Head in Site 1 is still complaining. If yes then I'll suggest to raise case with splunk support.

[jelizabeth]

thanks I'll give that a go