topic Re: Tried to add a search peer: Error while sending public key to search peer: No route to host in Reporting

Tried to add a search peer: Error while sending public key to search peer: No route to host

mhouse3 — Tue, 29 Aug 2017 17:21:42 GMT

(attempting 1 Indexer, +1 SH setup)

For some reason I am not able to add a search peer. I tried two approaches as follows:

Log into Splunk Web on the search head and click Settings at the top of the page. Click Distributed search in the Distributed Environment area. Click Search peers. On the Search peers page, select New. Specify the search peer, along with any authentication settings. Click Save. Repeat for each of the search head's search peers.

It produces: Encountered the following error while trying to save: Error while sending public key to search peer: No route to host

2) Ran the following from the command line producing the same result:
splunk add search-server https:// IndexerIPADDRES:8089 -auth admin:password -remoteUsername admin -remotePassword passremote

I am able to ping the machines from each respective machine as well as ping google.com from each of the machine.

I even checked the host level firewall and enable/permit the port in firewall using:
sudo firewall-cmd --zone=public --add-port=8089/tcp --permanent

and reload the config list:
firewall-cmd --list-all

Why I'm I getting this result?

Re: Tried to add a search peer: Error while sending public key to search peer: No route to host

jkat54 — Wed, 30 Aug 2017 18:03:13 GMT

can you successfully curl -k https://indexerIPAddress:8089 from the affected search peers? Or does that timeout?

If you get a timeout then you have a firewall or routing problem.

Re: Tried to add a search peer: Error while sending public key to search peer: No route to host

mhouse3 — Wed, 30 Aug 2017 18:53:15 GMT

I cannot.

I get the following:

curl: (7) Failed connect to 10.0.2.15:8089; No route to host

How to I resolve that?

Re: Tried to add a search peer: Error while sending public key to search peer: No route to host

woodcock — Mon, 04 Sep 2017 15:27:59 GMT

Your OS is telling you that the NIC that you are using doesn't have a network route that will enable them to reach that destination. Your choices are to send through a NIC that does, or add a route. If you are sysadmin on your host, then you can read this and figure it out (otherwise open a ticket to your networking team):
http://www.thegeekstuff.com/2012/04/route-examples

Re: Tried to add a search peer: Error while sending public key to search peer: No route to host

mhouse3 — Tue, 05 Sep 2017 15:16:42 GMT

Thank you woodcock. I will look into that.

Re: Tried to add a search peer: Error while sending public key to search peer: No route to host

woodcock — Sun, 10 Sep 2017 06:39:39 GMT

Be sure to come back and let us know what happened!

Re: Tried to add a search peer: Error while sending public key to search peer: No route to host

mhouse3 — Tue, 12 Sep 2017 00:07:27 GMT

I most certainly will.

Re: Tried to add a search peer: Error while sending public key to search peer: No route to host

jkat54 — Tue, 12 Sep 2017 12:07:36 GMT

Either a firewall is blocking the port or there is not a network route to the host. You need some basic network troubleshooting here.

Re: Tried to add a search peer: Error while sending public key to search peer: No route to host

mhouse3 — Tue, 12 Sep 2017 12:16:22 GMT

I figured as much.

So here is what I did:

The first thing I did was run nmap localhost -p 8000 just to see what I would get. I saw that it was open that that Splunkd is connected to it.

Then I ran nmap localhost -p 9997 and it says that it is closed on that local VM.

Perfect.

Next I run ince I am in a non-production environment.

Then I run nmap localhost -p 9997 and it still say says that it is closed on that local VM.

I troubleshoot I ran
And it shows that all the firewall is still up.

I go a step further then and ran
And it shows that the firewall isdown now.

Then I run nmap localhost -p 9997 and it still say says that it is closed on that local VM.

Re: Tried to add a search peer: Error while sending public key to search peer: No route to host

jkat54 — Tue, 12 Sep 2017 15:03:23 GMT

Is data receiving enabled on port 9997 for this indexer/peer?

Re: Tried to add a search peer: Error while sending public key to search peer: No route to host

mhouse3 — Sat, 16 Sep 2017 23:29:58 GMT

Woodstock I finally had time to check out the link you provided above. That link is not applicable to my problem set.

Here is why I say that. The link you provide provides instructions on how to add a route in Linux so that you have the ability for two machines (or more) to both be able to ping each machine as well as get to the external internet such as google.com. I had previously confirmed that I can ping every machine on my network as well as can get to the external network prior to taking the steps I identified in my question to add a search peer.

Re: Tried to add a search peer: Error while sending public key to search peer: No route to host

osmanysr — Thu, 03 Sep 2020 22:26:55 GMT

I had the same issue and I ran these commands on both the Search Head and the Indexer. And the communication started right away..

sudo firewall-cmd --zone=public --add-port=8089/tcp --permanent

sudo firewall-cmd --reload

sudo firewall-cmd --zone=public --list-all