Reporting

Tried to add a search peer: Error while sending public key to search peer: No route to host

mhouse3
Path Finder

(attempting 1 Indexer, +1 SH setup)

For some reason I am not able to add a search peer. I tried two approaches as follows:

  1. Log into Splunk Web on the search head and click Settings at the top of the page. Click Distributed search in the Distributed Environment area. Click Search peers. On the Search peers page, select New. Specify the search peer, along with any authentication settings. Click Save. Repeat for each of the search head's search peers.

It produces: Encountered the following error while trying to save: Error while sending public key to search peer: No route to host

2) Ran the following from the command line producing the same result:
splunk add search-server https:// IndexerIPADDRES:8089 -auth admin:password -remoteUsername admin -remotePassword passremote

I am able to ping the machines from each respective machine as well as ping google.com from each of the machine.

I even checked the host level firewall and enable/permit the port in firewall using:
sudo firewall-cmd --zone=public --add-port=8089/tcp --permanent

and reload the config list:
firewall-cmd --list-all

Why I'm I getting this result?

0 Karma

osmanysr
Engager

I had the same issue and I ran these commands on both the Search Head and the Indexer. And the communication started right away..

sudo firewall-cmd --zone=public --add-port=8089/tcp --permanent

sudo firewall-cmd --reload

sudo firewall-cmd --zone=public --list-all

0 Karma

woodcock
Esteemed Legend

Your OS is telling you that the NIC that you are using doesn't have a network route that will enable them to reach that destination. Your choices are to send through a NIC that does, or add a route. If you are sysadmin on your host, then you can read this and figure it out (otherwise open a ticket to your networking team):
http://www.thegeekstuff.com/2012/04/route-examples

0 Karma

mhouse3
Path Finder

Thank you woodcock. I will look into that.

0 Karma

woodcock
Esteemed Legend

Be sure to come back and let us know what happened!

0 Karma

mhouse3
Path Finder

I most certainly will.

0 Karma

mhouse3
Path Finder

Woodstock I finally had time to check out the link you provided above. That link is not applicable to my problem set.

Here is why I say that. The link you provide provides instructions on how to add a route in Linux so that you have the ability for two machines (or more) to both be able to ping each machine as well as get to the external internet such as google.com. I had previously confirmed that I can ping every machine on my network as well as can get to the external network prior to taking the steps I identified in my question to add a search peer.

0 Karma

jkat54
SplunkTrust
SplunkTrust

can you successfully curl -k https://indexerIPAddress:8089 from the affected search peers? Or does that timeout?

If you get a timeout then you have a firewall or routing problem.

0 Karma

mhouse3
Path Finder

I cannot.

I get the following:

curl: (7) Failed connect to 10.0.2.15:8089; No route to host

How to I resolve that?

0 Karma

jkat54
SplunkTrust
SplunkTrust

Either a firewall is blocking the port or there is not a network route to the host. You need some basic network troubleshooting here.

0 Karma

mhouse3
Path Finder

I figured as much.

So here is what I did:

The first thing I did was run nmap localhost -p 8000 just to see what I would get. I saw that it was open that that Splunkd is connected to it.

Then I ran nmap localhost -p 9997 and it says that it is closed on that local VM.

Perfect.

Next I run ince I am in a non-production environment.

Then I run nmap localhost -p 9997 and it still say says that it is closed on that local VM.

I troubleshoot I ran
And it shows that all the firewall is still up.

I go a step further then and ran
And it shows that the firewall isdown now.

Then I run nmap localhost -p 9997 and it still say says that it is closed on that local VM.

0 Karma

jkat54
SplunkTrust
SplunkTrust

Is data receiving enabled on port 9997 for this indexer/peer?

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...