Archive
Highlighted

Tried to add a search peer: Error while sending public key to search peer: No route to host

Path Finder

(attempting 1 Indexer, +1 SH setup)

For some reason I am not able to add a search peer. I tried two approaches as follows:

  1. Log into Splunk Web on the search head and click Settings at the top of the page. Click Distributed search in the Distributed Environment area. Click Search peers. On the Search peers page, select New. Specify the search peer, along with any authentication settings. Click Save. Repeat for each of the search head's search peers.

It produces: Encountered the following error while trying to save: Error while sending public key to search peer: No route to host

2) Ran the following from the command line producing the same result:
splunk add search-server https:// IndexerIPADDRES:8089 -auth admin:password -remoteUsername admin -remotePassword passremote

I am able to ping the machines from each respective machine as well as ping google.com from each of the machine.

I even checked the host level firewall and enable/permit the port in firewall using:
sudo firewall-cmd --zone=public --add-port=8089/tcp --permanent

and reload the config list:
firewall-cmd --list-all

Why I'm I getting this result?

0 Karma
Highlighted

Re: Tried to add a search peer: Error while sending public key to search peer: No route to host

SplunkTrust
SplunkTrust

can you successfully curl -k https://indexerIPAddress:8089 from the affected search peers? Or does that timeout?

If you get a timeout then you have a firewall or routing problem.

0 Karma
Highlighted

Re: Tried to add a search peer: Error while sending public key to search peer: No route to host

Path Finder

I cannot.

I get the following:

curl: (7) Failed connect to 10.0.2.15:8089; No route to host

How to I resolve that?

0 Karma
Highlighted

Re: Tried to add a search peer: Error while sending public key to search peer: No route to host

SplunkTrust
SplunkTrust

Either a firewall is blocking the port or there is not a network route to the host. You need some basic network troubleshooting here.

0 Karma
Highlighted

Re: Tried to add a search peer: Error while sending public key to search peer: No route to host

Path Finder

I figured as much.

So here is what I did:

The first thing I did was run nmap localhost -p 8000 just to see what I would get. I saw that it was open that that Splunkd is connected to it.

Then I ran nmap localhost -p 9997 and it says that it is closed on that local VM.

Perfect.

Next I run ince I am in a non-production environment.

Then I run nmap localhost -p 9997 and it still say says that it is closed on that local VM.

I troubleshoot I ran
And it shows that all the firewall is still up.

I go a step further then and ran
And it shows that the firewall isdown now.

Then I run nmap localhost -p 9997 and it still say says that it is closed on that local VM.

0 Karma
Highlighted

Re: Tried to add a search peer: Error while sending public key to search peer: No route to host

SplunkTrust
SplunkTrust

Is data receiving enabled on port 9997 for this indexer/peer?

0 Karma
Highlighted

Re: Tried to add a search peer: Error while sending public key to search peer: No route to host

Esteemed Legend

Your OS is telling you that the NIC that you are using doesn't have a network route that will enable them to reach that destination. Your choices are to send through a NIC that does, or add a route. If you are sysadmin on your host, then you can read this and figure it out (otherwise open a ticket to your networking team):
http://www.thegeekstuff.com/2012/04/route-examples

0 Karma
Highlighted

Re: Tried to add a search peer: Error while sending public key to search peer: No route to host

Path Finder

Thank you woodcock. I will look into that.

0 Karma
Highlighted

Re: Tried to add a search peer: Error while sending public key to search peer: No route to host

Esteemed Legend

Be sure to come back and let us know what happened!

0 Karma
Highlighted

Re: Tried to add a search peer: Error while sending public key to search peer: No route to host

Path Finder

I most certainly will.

0 Karma