Tried to add a search peer: Error while sending public key to search peer: No route to host

[mhouse3]

(attempting 1 Indexer, +1 SH setup)

For some reason I am not able to add a search peer. I tried two approaches as follows:

1. Log into Splunk Web on the search head and click Settings at the top of the page. Click Distributed search in the Distributed Environment area. Click Search peers. On the Search peers page, select New. Specify the search peer, along with any authentication settings. Click Save. Repeat for each of the search head's search peers.

It produces: Encountered the following error while trying to save: Error while sending public key to search peer: No route to host

2) Ran the following from the command line producing the same result:
splunk add search-server https:// IndexerIPADDRES:8089 -auth admin:password -remoteUsername admin -remotePassword passremote

I am able to ping the machines from each respective machine as well as ping google.com from each of the machine.

I even checked the host level firewall and enable/permit the port in firewall using:
sudo firewall-cmd --zone=public --add-port=8089/tcp --permanent

and reload the config list:
firewall-cmd --list-all

Why I'm I getting this result?

[osmanysr]

I had the same issue and I ran these commands on both the Search Head and the Indexer. And the communication started right away..

sudo firewall-cmd --zone=public --add-port=8089/tcp --permanent

sudo firewall-cmd --reload

sudo firewall-cmd --zone=public --list-all

[woodcock]

Your OS is telling you that the NIC that you are using doesn't have a network route that will enable them to reach that destination. Your choices are to send through a NIC that does, or add a route. If you are sysadmin on your host, then you can read this and figure it out (otherwise open a ticket to your networking team):
http://www.thegeekstuff.com/2012/04/route-examples

[mhouse3]

Thank you woodcock. I will look into that.

[woodcock]

Be sure to come back and let us know what happened!

[mhouse3]

I most certainly will.

[mhouse3]

Woodstock I finally had time to check out the link you provided above. That link is not applicable to my problem set.

Here is why I say that. The link you provide provides instructions on how to add a route in Linux so that you have the ability for two machines (or more) to both be able to ping each machine as well as get to the external internet such as google.com. I had previously confirmed that I can ping every machine on my network as well as can get to the external network prior to taking the steps I identified in my question to add a search peer.

[jkat54]

can you successfully curl -k https://indexerIPAddress:8089 from the affected search peers? Or does that timeout?

If you get a timeout then you have a firewall or routing problem.

[mhouse3]

I cannot.

I get the following:

curl: (7) Failed connect to 10.0.2.15:8089; No route to host

How to I resolve that?

[jkat54]

Either a firewall is blocking the port or there is not a network route to the host. You need some basic network troubleshooting here.

[mhouse3]

I figured as much.

So here is what I did:

The first thing I did was run nmap localhost -p 8000 just to see what I would get. I saw that it was open that that Splunkd is connected to it.

Then I ran nmap localhost -p 9997 and it says that it is closed on that local VM.

Perfect.

Next I run ince I am in a non-production environment.

Then I run nmap localhost -p 9997 and it still say says that it is closed on that local VM.

I troubleshoot I ran
And it shows that all the firewall is still up.

I go a step further then and ran
And it shows that the firewall isdown now.

Then I run nmap localhost -p 9997 and it still say says that it is closed on that local VM.

[jkat54]

Is data receiving enabled on port 9997 for this indexer/peer?