Deployment Architecture

Why am I getting the following "send failure" message in my internal logs: "pushing PK to search peer" ?

jelizabeth
New Member

Here is the complete warning message:

Send failure while pushing PK to search peer = https://*.*.*.*:8089 , Read Timeout

I'm getting the above warning messages in the internal Splunk logs every minute from each of our 3 search heads.

The search peer in question is in our secondary site (let's say B) to the search heads (site A), but there are two other search peers in the same site (B) which we don't get any warning messages for.

I've done a ping and netcat from each of the search heads in site A to each of the three search peers in Site B and the results are the same for each one, connection established and similar ping times.

It's not a connection issue, so i'm wondering what else could be causing it?

0 Karma

harsmarvania57
Ultra Champion

It looks like you have some network issues between site A and site B (Maybe high latency). Same problem faced by other user previously and for them it was network issue. (reference : https://answers.splunk.com/answers/455635/why-is-my-search-head-cluster-not-working-after-up-1.html)

0 Karma

jelizabeth
New Member

As previously stated we don't believe it's a network issue as all tests between instances show no latency. we are looking for an alternative reason as to what could be causing the issue.

0 Karma

harsmarvania57
Ultra Champion

In that case you can directly distribute key files using process given here in Splunk Docs and after that check again whether splunk on Search Head in Site 1 is still complaining. If yes then I'll suggest to raise case with splunk support.

0 Karma

jelizabeth
New Member

thanks I'll give that a go

0 Karma
Get Updates on the Splunk Community!

Why You Can't Miss .conf25: Unleashing the Power of Agentic AI with Splunk & Cisco

The Defining Technology Movement of Our Lifetime The advent of agentic AI is arguably the defining technology ...

Deep Dive into Federated Analytics: Unlocking the Full Power of Your Security Data

In today’s complex digital landscape, security teams face increasing pressure to protect sprawling data across ...

Your summer travels continue with new course releases

Summer in the Northern hemisphere is in full swing, and is often a time to travel and explore. If your summer ...