Deployment Architecture

How to properly connect a search head cluster to a search peer?

Engager

I'm having a very hard time connecting my search head cluster to my search peer. I have stepped through the search head documentation very carefully located here: http://docs.splunk.com/Documentation/Splunk/6.5.0/DistSearch/SHCdeploymentoverview
I have successfully installed my deployer and added the [shclustering] stanza to the /opt/splunk/etc/system/local/server.conf file and added the pass4SymmKey and shcluster_label.

I then ran splunk init shcluster-config on each of my search head members and restarted Splunk. Each one ran successfully without any reported errors. I'm also able to run splunk bootstrap shcluster-captain without any issues and splunk show shcluster-status doesn't report any problems:

[splunk@lelsplunksh02 ~]$ splunk show shcluster-status

 Captain:
                          dynamic_captain : 1
                          elected_captain : Thu Oct 13 15:48:05 2016
                                       id : C2403815-55A2-413E-AF26-4998CFD9508F
                         initialized_flag : 1
                                    label : lelsplunksh03
                         maintenance_mode : 0
                                 mgmt_uri : https://splunkserver:8089
                    min_peers_joined_flag : 1
                     rolling_restart_flag : 0
                       service_ready_flag : 1

 Members:
        lelsplunksh02
                                    label : lelsplunksh02
                                 mgmt_uri : https://splunkserver:8089
                           mgmt_uri_alias : https://xx.xxx.xx.xxx:8089
                                   status : Up
        lelsplunksh04
                                    label : lelsplunksh04
                                 mgmt_uri : https://splunkserver:8089
                           mgmt_uri_alias : https://xx.xxx.xx.xxx:8089
                                   status : Up
        lelsplunksh03
                                    label : lelsplunksh03
                                 mgmt_uri : https://splunkserver:8089
                           mgmt_uri_alias : https://xx.xxx.xx.xxx:8089
                                   status : Up

My problem starts when I try to add my search peer. I only have one indexer and I'm following this doc: http://docs.splunk.com/Documentation/Splunk/6.5.0/DistSearch/Connectclustersearchheadstosearchpeers

I'm running:

splunk add search-server https://splunkserver:8089 -auth admin:pswd -remoteUsername admin -remotePassword pswd

This also runs successfully, but I'm just not getting any results when I connect to my search head and run a search. I can run the exact same search on the indexer itself and it returns results. I can't see any errors in logs on either the indexer or the search head members.

Any help would be appreciated to point me in the right direction.

0 Karma
1 Solution

SplunkTrust
SplunkTrust

Hi there zipmaster,

first of I had my own problems with searchhead <-> indexer connection. It's easy to make a mistake here.


After the execution of the command there should be a distsearch.conf in the $SplunkHome/etc/system/local

Could you tell me if there is one?
If yes, could you maybe post it's content here, too?

Thanks!
pyro_wood

(PS: in approx. 1.5 h I'm at work, so I will post you parts of my own guide where I tried to setup distributed search)


EDIT:

If you would like to delete existing cluster config on a search head (to beginn from start) do the following:

On every SH do the following commands:
splunk remove shcluster-member
(wait approx. 1 minute)
splunk stop
splunk clean all
splunk start

Now you should have clean SH's without cluster config.

Initiate SH-Cluster config:
Go on every SH in server.conf and post the following (alter the config for every sh)

[shclustering]
confdeployfetchurl = https://deployer:8089
disabled = 0
mgmt
uri = https://sh1:8089
pass4SymmKey = e.g.:splunkisawesome
shclusterlabel = e.g.:SH-Cluster1

restart splunk afterwards
restart splunk

Initialize Cluster-Captain with this command:
splunk bootstrap shcluster-captain -servers_list "https://sh1:8089,https://sh2:8089,https://sh3:8089"

(it takes a while)

then do:
splunk show shcluster-status


Next steps:
Go on every Search-Head and create a Stanza called [clustering] in server.conf:

[clustering]
search_server= https://indexer:8089
mode = searchhead
pass4SymmKey = e.g.:splunkisawesome

Then execute:
restart splunk

Try it out! Sometime this does the trick already.


If not... and I don't know why this only happens occasionaly do these steps as well:

Now you need to setup authentication for the Indexers:

Copy via scp (or other) every "trusted.pem" from every SH:
/opt/splunk/etc/auth/distServerKeys/trusted.pem

to the indexers into the corresponding file:
$SplunkHome/etc/auth/distServerKeys//
$SplunkHome/etc/auth/distServerKeys//
$SplunkHome/etc/auth/distServerKeys//

(if those directories arent there create them)

Restart indexer
splunk restart

HOPE THIS HELPS 😉
Just ask, if you have any further questions.

View solution in original post