Why am I unable to set up a search head cluster and getting error "Failed to bootstrap this node as a captain"?

[wangsimingxaxis]

I am trying to set up a search head cluster, but failed.

Below are my settings:
1) on Search head1 (xx.xx.xx.aa)
run below command then restart splunk

splunk init shcluster-config -auth admin:changeme -mgmt_uri https://xx.xx.xx.aa:8089 -replication_port 8888 -replication_factor 2 -conf_deploy_fetch_url https://xx.xx.xx.cc:8089 -secret changeme -shcluster_label shcluster1

2) on Search head2 (xx.xx.xx.bb)
run below command then restart splunk

splunk init shcluster-config -auth admin:changeme -mgmt_uri https://xx.xx.xx.bb:8089 -replication_port 8888 -replication_factor 2 -conf_deploy_fetch_url https://xx.xx.xx.cc:8089 -secret changeme -shcluster_label shcluster1

3) on Deployer host (xx.xx.xx.cc), set below in server.conf

[shclustering]
shcluster_label = shcluster1

4) Bring up captain on SH1

/opt/splunk/bin/splunk bootstrap shcluster-captain -servers_list "https://xx.xx.xx.aa:8089,https://xx.xx.xx.bb:8089" -auth admin:changeme

I'm getting the error below:

[root@splunksh1hk1 ~]# tail -f /opt/splunk/var/log/splunk/splunkd.log
06-01-2016 20:50:48.977 +0000 INFO  KeyManagerLocalhost - Public key already exists: /opt/splunk/etc/auth/distServerKeys/trusted.pem
06-01-2016 20:50:48.977 +0000 INFO  KeyManagerLocalhost - Reading public key for localhost: /opt/splunk/etc/auth/distServerKeys/trusted.pem
06-01-2016 20:50:48.977 +0000 INFO  KeyManagerLocalhost - Finished reading public key for localhost: /opt/splunk/etc/auth/distServerKeys/trusted.pem
06-01-2016 20:50:48.977 +0000 INFO  KeyManagerLocalhost - Reading private key for localhost: /opt/splunk/etc/auth/distServerKeys/private.pem
06-01-2016 20:50:48.977 +0000 INFO  KeyManagerLocalhost - Finished reading private key for localhost: /opt/splunk/etc/auth/distServerKeys/private.pem
06-01-2016 20:50:48.984 +0000 INFO  ServerConfig - Using REMOTE_SERVER_NAME=5453F6EB-0F41-49FA-9203-F6A6FAED2D85
06-01-2016 20:50:48.987 +0000 INFO  ServerRoles - Declared role=search_head.
06-01-2016 20:51:44.987 +0000 INFO  DatabaseDirectoryManager - idx=_internal Writing a bucket manifest in hotWarmPath='/opt/splunk/var/lib/splunk/_internaldb/db', pendingBucketUpdates=0 .  Reason='Buckets were rebuilt or tsidx-minified (bucket_count=1).'
06-01-2016 20:53:43.271 +0000 ERROR SHCRaftConsensus - Failed to bootstrap this node as a captain.
06-01-2016 21:00:23.340 +0000 ERROR SHCRaftConsensus - Failed to bootstrap this node as a captain.

[ThomasControlwa]

you need min 3 member of SH clustering,
also the member needs to be the same setup. the context of 3 member is to collect a "Captain" role of your shcluster.

make sure your member can connect to the clustermaster and deployer, additional to the deploymentserver

try to fix your quest as following:
setup a new SH member
clean your xx/local/server.conf with all [shcluster] content
only on your SH MEMBER!

-->rm -rf SPLUNK_HOME /etc/instance.cfg
-->rm -rf /SPLUNK_HOME/var/run/splunk/_raft/*
--> ./splunk restart

now build your fresh SH Cluster with
on all sh member
-->./splunk edit cluster-config -master_uri https://xx.xx.xx.xx:8089 -mode slave -site site1 -replication_port 8080 -secret your choice
restart

-->./splunk init shcluster-config -mgmt_uri https://your.sh.member.DNS or IP:8089 -replication_port 8080 -secret changed
-->./splunk bootstrap shcluster-captain -servers_list https://member1:8089,https://member2:8089,https://member3:8089

don't forget to click solfed 🙂

additional links:
http://docs.splunk.com/Documentation/Splunk/7.0.0/DistSearch/SHCconfigurationoverview
hope it helps

[dbroggy]

I don't understand why Thomas is using the cluster-config command to add a search head. should be the shcluster-config command.

[ddrillic]

@dbroggy, it's an interesting thing, we configure using the cluster-config parameter to set the search head as part of the indexer cluster and then we use the shcluster-config parameter to set it up as part of the search head cluster.

So, the node is part of the indexer cluster and also as part of the sub-cluster of the search head cluster.

[dbroggy]

But I'm pretty sure that's the wrong syntax.
"-mode slave" assigns an indexer to the indexer cluster.
And search heads don't do replication to an index cluster so replication_port makes no sense.
The command should be:
./splunk edit cluster-config -mode searchhead -site site0 -master_uri https://:8089 -secret pass123
Ref:
https://docs.splunk.com/Documentation/Splunk/6.4.1/DistSearch/SHCandindexercluster

[amielke]

Is the -site option necessary or can I omitting this?

[ThomasControlwa]

Hi,
I think it depends If your Cluster Is a multisite Cluster, that means you work wit site1 and site2.
If you are not sure, post your SH-Cluster conf
cheers

[amielke]

Oh, we use a single site or say no site Cluster.
Is this in this case necessary?

[ThomasControlwa]

no, you don't Need this. please take a look @ here http://docs.splunk.com/Documentation/Splunk/7.0.2/Indexer/ConfiguresearchheadwithCLI

Edit the search head settings

You can also use the CLI to edit the configuration later.

Important: When you first enable a search head, you use the splunk edit cluster-config command. To change the search head configuration, you must instead use the splunk edit cluster-master command.

For example, to change the security key (secret), use this command:

splunk edit cluster-master https://10.160.31.200:8089 -secret newsecret123

Important: The splunk edit cluster-master command always takes the current master URI:port value as its initial parameter. For example, this command connects the search head to a different master by setting a new value for the -master_uri parameter, but it provides the value for the old master as its initial parameter:

splunk edit cluster-master https://10.160.31.200:8089 -master_uri https://10.160.31.55:8089

Refer to the CLI clustering help, along with the server.conf specification file, for the list of configurable settings.

[ThomasControlwa]

please rate my answer 😄 many thanks and cheers

[dolivasoh]

You cannot achieve a quorum with only two search heads. Add another.