Deployment Architecture

How to properly connect a search head cluster to a search peer?


I'm having a very hard time connecting my search head cluster to my search peer. I have stepped through the search head documentation very carefully located here:
I have successfully installed my deployer and added the [shclustering] stanza to the /opt/splunk/etc/system/local/server.conf file and added the pass4SymmKey and shcluster_label.

I then ran splunk init shcluster-config on each of my search head members and restarted Splunk. Each one ran successfully without any reported errors. I'm also able to run splunk bootstrap shcluster-captain without any issues and splunk show shcluster-status doesn't report any problems:

[splunk@lelsplunksh02 ~]$ splunk show shcluster-status

                          dynamic_captain : 1
                          elected_captain : Thu Oct 13 15:48:05 2016
                                       id : C2403815-55A2-413E-AF26-4998CFD9508F
                         initialized_flag : 1
                                    label : lelsplunksh03
                         maintenance_mode : 0
                                 mgmt_uri : https://splunkserver:8089
                    min_peers_joined_flag : 1
                     rolling_restart_flag : 0
                       service_ready_flag : 1

                                    label : lelsplunksh02
                                 mgmt_uri : https://splunkserver:8089
                           mgmt_uri_alias :
                                   status : Up
                                    label : lelsplunksh04
                                 mgmt_uri : https://splunkserver:8089
                           mgmt_uri_alias :
                                   status : Up
                                    label : lelsplunksh03
                                 mgmt_uri : https://splunkserver:8089
                           mgmt_uri_alias :
                                   status : Up

My problem starts when I try to add my search peer. I only have one indexer and I'm following this doc:

I'm running:

splunk add search-server https://splunkserver:8089 -auth admin:pswd -remoteUsername admin -remotePassword pswd

This also runs successfully, but I'm just not getting any results when I connect to my search head and run a search. I can run the exact same search on the indexer itself and it returns results. I can't see any errors in logs on either the indexer or the search head members.

Any help would be appreciated to point me in the right direction.

0 Karma
1 Solution


Hi there zipmaster,

first of I had my own problems with searchhead <-> indexer connection. It's easy to make a mistake here.

After the execution of the command there should be a distsearch.conf in the $SplunkHome/etc/system/local

Could you tell me if there is one?
If yes, could you maybe post it's content here, too?


(PS: in approx. 1.5 h I'm at work, so I will post you parts of my own guide where I tried to setup distributed search)


If you would like to delete existing cluster config on a search head (to beginn from start) do the following:

On every SH do the following commands:
splunk remove shcluster-member
(wait approx. 1 minute)
splunk stop
splunk clean all
splunk start

Now you should have clean SH's without cluster config.

Initiate SH-Cluster config:
Go on every SH in server.conf and post the following (alter the config for every sh)

conf_deploy_fetch_url = https://deployer:8089
disabled = 0
mgmt_uri = https://sh1:8089
pass4SymmKey = e.g.:splunkisawesome
shcluster_label = e.g.:SH-Cluster_1

restart splunk afterwards
restart splunk

Initialize Cluster-Captain with this command:
splunk bootstrap shcluster-captain -servers_list "https://sh1:8089,https://sh2:8089,https://sh3:8089"

(it takes a while)

then do:
splunk show shcluster-status

Next steps:
Go on every Search-Head and create a Stanza called [clustering] in server.conf:

search_server= https://indexer:8089
mode = searchhead
pass4SymmKey = e.g.:splunkisawesome

Then execute:
restart splunk

Try it out! Sometime this does the trick already.

If not... and I don't know why this only happens occasionaly do these steps as well:

Now you need to setup authentication for the Indexers:

Copy via scp (or other) every "trusted.pem" from every SH:

to the indexers into the corresponding file:

(if those directories arent there create them)

Restart indexer
splunk restart

Just ask, if you have any further questions.

View solution in original post