I am trying to set up a search head cluster, but failed.
Below are my settings:
1) on Search head1 (xx.xx.xx.aa)
run below command then restart splunk
splunk init shcluster-config -auth admin:changeme -mgmt_uri https://xx.xx.xx.aa:8089 -replication_port 8888 -replication_factor 2 -conf_deploy_fetch_url https://xx.xx.xx.cc:8089 -secret changeme -shcluster_label shcluster1
2) on Search head2 (xx.xx.xx.bb)
run below command then restart splunk
splunk init shcluster-config -auth admin:changeme -mgmt_uri https://xx.xx.xx.bb:8089 -replication_port 8888 -replication_factor 2 -conf_deploy_fetch_url https://xx.xx.xx.cc:8089 -secret changeme -shcluster_label shcluster1
3) on Deployer host (xx.xx.xx.cc), set below in server.conf
[shclustering] shcluster_label = shcluster1
4) Bring up captain on SH1
/opt/splunk/bin/splunk bootstrap shcluster-captain -servers_list "https://xx.xx.xx.aa:8089,https://xx.xx.xx.bb:8089" -auth admin:changeme
I'm getting the error below:
[root@splunksh1hk1 ~]# tail -f /opt/splunk/var/log/splunk/splunkd.log 06-01-2016 20:50:48.977 +0000 INFO KeyManagerLocalhost - Public key already exists: /opt/splunk/etc/auth/distServerKeys/trusted.pem 06-01-2016 20:50:48.977 +0000 INFO KeyManagerLocalhost - Reading public key for localhost: /opt/splunk/etc/auth/distServerKeys/trusted.pem 06-01-2016 20:50:48.977 +0000 INFO KeyManagerLocalhost - Finished reading public key for localhost: /opt/splunk/etc/auth/distServerKeys/trusted.pem 06-01-2016 20:50:48.977 +0000 INFO KeyManagerLocalhost - Reading private key for localhost: /opt/splunk/etc/auth/distServerKeys/private.pem 06-01-2016 20:50:48.977 +0000 INFO KeyManagerLocalhost - Finished reading private key for localhost: /opt/splunk/etc/auth/distServerKeys/private.pem 06-01-2016 20:50:48.984 +0000 INFO ServerConfig - Using REMOTE_SERVER_NAME=5453F6EB-0F41-49FA-9203-F6A6FAED2D85 06-01-2016 20:50:48.987 +0000 INFO ServerRoles - Declared role=search_head. 06-01-2016 20:51:44.987 +0000 INFO DatabaseDirectoryManager - idx=_internal Writing a bucket manifest in hotWarmPath='/opt/splunk/var/lib/splunk/_internaldb/db', pendingBucketUpdates=0 . Reason='Buckets were rebuilt or tsidx-minified (bucket_count=1).' 06-01-2016 20:53:43.271 +0000 ERROR SHCRaftConsensus - Failed to bootstrap this node as a captain. 06-01-2016 21:00:23.340 +0000 ERROR SHCRaftConsensus - Failed to bootstrap this node as a captain.
you need min 3 member of SH clustering,
also the member needs to be the same setup. the context of 3 member is to collect a "Captain" role of your shcluster.
make sure your member can connect to the clustermaster and deployer, additional to the deploymentserver
try to fix your quest as following:
setup a new SH member
clean your xx/local/server.conf with all [shcluster] content
only on your SH MEMBER!
-->rm -rf SPLUNKHOME /etc/instance.cfg
-->rm -rf /SPLUNKHOME/var/run/splunk/_raft/*
--> ./splunk restart
now build your fresh SH Cluster with
on all sh member
-->./splunk edit cluster-config -masteruri https://xx.xx.xx.xx:8089 -mode slave -site site1 -replicationport 8080 -secret your choice
-->./splunk init shcluster-config -mgmturi https://your.sh.member.DNS or IP:8089 -replicationport 8080 -secret changed
-->./splunk bootstrap shcluster-captain -servers_list https://member1:8089,https://member2:8089,https://member3:8089
don't forget to click solfed 🙂
hope it helps
@dbroggy, it's an interesting thing, we configure using the
cluster-config parameter to set the search head as part of the indexer cluster and then we use the
shcluster-config parameter to set it up as part of the search head cluster.
So, the node is part of the indexer cluster and also as part of the sub-cluster of the search head cluster.
But I'm pretty sure that's the wrong syntax.
"-mode slave" assigns an indexer to the indexer cluster.
And search heads don't do replication to an index cluster so replicationport makes no sense.
The command should be:
./splunk edit cluster-config -mode searchhead -site site0 -masteruri https://:8089 -secret pass123
no, you don't Need this. please take a look @ here http://docs.splunk.com/Documentation/Splunk/7.0.2/Indexer/ConfiguresearchheadwithCLI
Edit the search head settings
You can also use the CLI to edit the configuration later.
Important: When you first enable a search head, you use the splunk edit cluster-config command. To change the search head configuration, you must instead use the splunk edit cluster-master command.
For example, to change the security key (secret), use this command:
splunk edit cluster-master https://10.160.31.200:8089 -secret newsecret123
Important: The splunk edit cluster-master command always takes the current master URI:port value as its initial parameter. For example, this command connects the search head to a different master by setting a new value for the -master_uri parameter, but it provides the value for the old master as its initial parameter:
Refer to the CLI clustering help, along with the server.conf specification file, for the list of configurable settings.