I'm having a very hard time connecting my search head cluster to my search peer. I have stepped through the search head documentation very carefully located here: http://docs.splunk.com/Documentation/Splunk/6.5.0/DistSearch/SHCdeploymentoverview
I have successfully installed my deployer and added the
[shclustering] stanza to the /opt/splunk/etc/system/local/server.conf file and added the pass4SymmKey and shcluster_label.
I then ran
splunk init shcluster-config on each of my search head members and restarted Splunk. Each one ran successfully without any reported errors. I'm also able to run
splunk bootstrap shcluster-captain without any issues and
splunk show shcluster-status doesn't report any problems:
[splunk@lelsplunksh02 ~]$ splunk show shcluster-status Captain: dynamic_captain : 1 elected_captain : Thu Oct 13 15:48:05 2016 id : C2403815-55A2-413E-AF26-4998CFD9508F initialized_flag : 1 label : lelsplunksh03 maintenance_mode : 0 mgmt_uri : https://splunkserver:8089 min_peers_joined_flag : 1 rolling_restart_flag : 0 service_ready_flag : 1 Members: lelsplunksh02 label : lelsplunksh02 mgmt_uri : https://splunkserver:8089 mgmt_uri_alias : https://xx.xxx.xx.xxx:8089 status : Up lelsplunksh04 label : lelsplunksh04 mgmt_uri : https://splunkserver:8089 mgmt_uri_alias : https://xx.xxx.xx.xxx:8089 status : Up lelsplunksh03 label : lelsplunksh03 mgmt_uri : https://splunkserver:8089 mgmt_uri_alias : https://xx.xxx.xx.xxx:8089 status : Up
My problem starts when I try to add my search peer. I only have one indexer and I'm following this doc: http://docs.splunk.com/Documentation/Splunk/6.5.0/DistSearch/Connectclustersearchheadstosearchpeers
splunk add search-server https://splunkserver:8089 -auth admin:pswd -remoteUsername admin -remotePassword pswd
This also runs successfully, but I'm just not getting any results when I connect to my search head and run a search. I can run the exact same search on the indexer itself and it returns results. I can't see any errors in logs on either the indexer or the search head members.
Any help would be appreciated to point me in the right direction.
Hi there zipmaster,
first of I had my own problems with searchhead <-> indexer connection. It's easy to make a mistake here.
After the execution of the command there should be a distsearch.conf in the $SplunkHome/etc/system/local
Could you tell me if there is one?
If yes, could you maybe post it's content here, too?
(PS: in approx. 1.5 h I'm at work, so I will post you parts of my own guide where I tried to setup distributed search)
If you would like to delete existing cluster config on a search head (to beginn from start) do the following:
On every SH do the following commands:
splunk remove shcluster-member
(wait approx. 1 minute)
splunk clean all
Now you should have clean SH's without cluster config.
Initiate SH-Cluster config:
Go on every SH in server.conf and post the following (alter the config for every sh)
restart splunk afterwards
Initialize Cluster-Captain with this command:
splunk bootstrap shcluster-captain -servers_list "https://sh1:8089,https://sh2:8089,https://sh3:8089"
(it takes a while)
splunk show shcluster-status
Go on every Search-Head and create a Stanza called [clustering] in server.conf:
mode = searchhead
pass4SymmKey = e.g.:splunkisawesome
Try it out! Sometime this does the trick already.
If not... and I don't know why this only happens occasionaly do these steps as well:
Now you need to setup authentication for the Indexers:
Copy via scp (or other) every "trusted.pem" from every SH:
to the indexers into the corresponding file:
(if those directories arent there create them)
HOPE THIS HELPS 😉
Just ask, if you have any further questions.