Deployment Architecture

Timeout on configuring Distributed Search

New Member

I am setting up Distributed Search for the first time. When I enter the Peer and Authentication information, I get the red error msg;

Encountered the following error while trying to save: In handler 'distsearch-peer': Error while sending public key to search peer https://splunk11:8089: Connect timed out.

Splunk is running on the peer, but when I try to manually telnet to 8089, the connection is immediately refused. When I do a netstat -an on the box, it is not showing anything listening on 8089.

0 Karma


Troubleshooting Steps:

1.Login to Splunk indexer & Splunk search head (ssh)
2.From indexer : Check splunk service is running : $SPLUNK_HOME/bin/splunk status

Possible Error:
splunkd xxxx was not running.
Stopping splunk helpers...
couldn't send SIGTERM to pid xxxx: Operation not permitted
Couldn't send SIGTERM to some splunk helpers.              [FAILED]
Error: Unable to stop splunk helpers.
  1. From indexer : Goto >> $SPLUNK_HOME/var/run/splunk/

    $SPLUNK_HOME/bin/splunk status
    $SPLUNK_HOME/bin/splunk start

  2. It should work.

Possibly Firewall issues:

From indexer :

telnet localhost 8089

Check 8089 port is listening

From search head:

telnet splunk-indexer 8089

check u can able to reach 8089 port of indexer
You should have allowed in firewall rules.

0 Karma


I noticed on mine that the new search head was listening on 8089 but the older indexer was listening on 8090 (just FYI)

0 Karma


remote peer not listening on 8089? sounds wired, restart it and check if splunkd 8089 is opened

0 Karma

New Member

Splunk was configured to listen on a different/non-standard port. Once I solved that, I was able to get the Instances talking to each other.

0 Karma
Register for .conf21 Now! Go Vegas or Go Virtual!

How will you .conf21? You decide! Go in-person in Las Vegas, 10/18-10/21, or go online with .conf21 Virtual, 10/19-10/20.