Tried to add a search peer: Error while sending public key to search peer: Connection closed by peer

[Jarohnimo]

(attempting 1 Indexer, +1 SH setup)

Tried the Following the Instructions from Splunk
1. Log into Splunk Web on the search head and click Settings at the top of the page.

1. Click Distributed search in the Distributed Environment area.

2. Click Search peers.

3. On the Search peers page, select New.

4. Specify the search peer, along with any authentication settings.

Note: You must precede the search peer's host name or IP address with the URI scheme, either "http" or "https".

1. Click Save.

   1. Repeat for each of the search head's search peers.

I've confirmed that my management port is open. Was able to confirm that all my forwarders are also connecting via 8089 so i'm not sure why it's saying connection closed ... The search peer has everything on it.

It produces: Encountered the following error while trying to save: In handler 'distsearch-peer': Error while sending public key to search peer: Connection closed by peer

I've also tried to add make this search head a slave to my indexer that is the license master but when I add http://myserver.domain.com:8089 I get similar results: Bad Request — In handler 'localslave': editTracker failed, reason='Unable to connect to license master: http://myserver.domain.com:8089 Unknown read error'

This is wild because I'm able to telnet fine, when I netstat it shows connection to 8089 on the search peer, fire wall is wide open but still no connection error... Is there any steps I have to do prior? Does this only work with https? Are there any steps other than changing the default password on both nodes that I have to do? Do I need to adjust anything in the DMC prior? or perhaps in any of the config files to prep? Thanks for your help

[neelamsantosh]

Check the host level firewall. as often we avoid it in our troubleshooting.
enable/permit the port in firewall using:
sudo firewall-cmd --zone=public --add-port=8089/tcp --permanent

and reload the config list:
firewall-cmd --list-all

Now configure the search peers.

[Jarohnimo · Accepted solution]

I Figured it out, Both messages were related to me just not knowing what the heck i'm doing.

Licensing issues: The account that i was logged in with was an Admin in Splunk on both nodes but didn't have any rights on the server. I then tried switching the Search HEad to Slave with my Server Admin account (Also splunk admin) and then it worked. I found one of my issues was on the new server (SH) i jumped the gun adding the license to this box and then splunk got mad. You only need it on the master.

Adding the Distributive search answer:

I was typing http://myserver.domain.com:8089 (Even though its an option that suppose to work) for me it didn't. Just go with their first option in: Servername:Port# it will work. (Remember to use an account that has local admin rights on both boxes as well is a splunk admin... preferably a domain account)

[anand_singh17]

Sometimes the service is not reachable, so gives this error.

You can use inbuilt user account or domain account.