Deployment Architecture

Timeout on configuring Distributed Search

robtheorg
New Member

I am setting up Distributed Search for the first time. When I enter the Peer and Authentication information, I get the red error msg;

Encountered the following error while trying to save: In handler 'distsearch-peer': Error while sending public key to search peer https://splunk11:8089: Connect timed out.

Splunk is running on the peer, but when I try to manually telnet to 8089, the connection is immediately refused. When I do a netstat -an on the box, it is not showing anything listening on 8089.

0 Karma

splunker12er
Motivator

Troubleshooting Steps:

1.Login to Splunk indexer & Splunk search head (ssh)
2.From indexer : Check splunk service is running : $SPLUNK_HOME/bin/splunk status

Possible Error:
splunkd xxxx was not running.
Stopping splunk helpers...
couldn't send SIGTERM to pid xxxx: Operation not permitted
Couldn't send SIGTERM to some splunk helpers.              [FAILED]
Error: Unable to stop splunk helpers.
  1. From indexer : Goto >> $SPLUNK_HOME/var/run/splunk/

    rm splunkd.pid
    $SPLUNK_HOME/bin/splunk status
    $SPLUNK_HOME/bin/splunk start

  2. It should work.

OR
Possibly Firewall issues:

From indexer :

telnet localhost 8089

Check 8089 port is listening

From search head:

telnet splunk-indexer 8089

check u can able to reach 8089 port of indexer
You should have allowed in firewall rules.

0 Karma

ldgrube
Engager

I noticed on mine that the new search head was listening on 8089 but the older indexer was listening on 8090 (just FYI)

0 Karma

hjwang
Contributor

remote peer not listening on 8089? sounds wired, restart it and check if splunkd 8089 is opened

0 Karma

robtheorg
New Member

Splunk was configured to listen on a different/non-standard port. Once I solved that, I was able to get the Instances talking to each other.

0 Karma
Get Updates on the Splunk Community!

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...