Timeout on configuring Distributed Search

[robtheorg]

I am setting up Distributed Search for the first time. When I enter the Peer and Authentication information, I get the red error msg;

Encountered the following error while trying to save: In handler 'distsearch-peer': Error while sending public key to search peer https://splunk11:8089: Connect timed out.

Splunk is running on the peer, but when I try to manually telnet to 8089, the connection is immediately refused. When I do a netstat -an on the box, it is not showing anything listening on 8089.

[splunker12er]

Troubleshooting Steps:

1.Login to Splunk indexer & Splunk search head (ssh)
2.From indexer : Check splunk service is running : $SPLUNK_HOME/bin/splunk status

Possible Error:
splunkd xxxx was not running.
Stopping splunk helpers...
couldn't send SIGTERM to pid xxxx: Operation not permitted
Couldn't send SIGTERM to some splunk helpers.              [FAILED]
Error: Unable to stop splunk helpers.

1. From indexer : Goto >> $SPLUNK_HOME/var/run/splunk/

   rm splunkd.pid
   $SPLUNK_HOME/bin/splunk status
   $SPLUNK_HOME/bin/splunk start

2. It should work.

OR
Possibly Firewall issues:

From indexer :

telnet localhost 8089

Check 8089 port is listening

From search head:

telnet splunk-indexer 8089

check u can able to reach 8089 port of indexer
You should have allowed in firewall rules.

[ldgrube]

I noticed on mine that the new search head was listening on 8089 but the older indexer was listening on 8090 (just FYI)

[hjwang]

remote peer not listening on 8089? sounds wired, restart it and check if splunkd 8089 is opened

[robtheorg]

Splunk was configured to listen on a different/non-standard port. Once I solved that, I was able to get the Instances talking to each other.