Deployment Architecture

Why am I unable to set up a search head cluster and getting error "Failed to bootstrap this node as a captain"?

wangsimingxaxis
Explorer

I am trying to set up a search head cluster, but failed.

Below are my settings:
1) on Search head1 (xx.xx.xx.aa)
run below command then restart splunk

splunk init shcluster-config -auth admin:changeme -mgmt_uri https://xx.xx.xx.aa:8089 -replication_port 8888 -replication_factor 2 -conf_deploy_fetch_url https://xx.xx.xx.cc:8089 -secret changeme -shcluster_label shcluster1

2) on Search head2 (xx.xx.xx.bb)
run below command then restart splunk

splunk init shcluster-config -auth admin:changeme -mgmt_uri https://xx.xx.xx.bb:8089 -replication_port 8888 -replication_factor 2 -conf_deploy_fetch_url https://xx.xx.xx.cc:8089 -secret changeme -shcluster_label shcluster1

3) on Deployer host (xx.xx.xx.cc), set below in server.conf

[shclustering]
shcluster_label = shcluster1

4) Bring up captain on SH1

/opt/splunk/bin/splunk bootstrap shcluster-captain -servers_list "https://xx.xx.xx.aa:8089,https://xx.xx.xx.bb:8089" -auth admin:changeme

I'm getting the error below:

[root@splunksh1hk1 ~]# tail -f /opt/splunk/var/log/splunk/splunkd.log
06-01-2016 20:50:48.977 +0000 INFO  KeyManagerLocalhost - Public key already exists: /opt/splunk/etc/auth/distServerKeys/trusted.pem
06-01-2016 20:50:48.977 +0000 INFO  KeyManagerLocalhost - Reading public key for localhost: /opt/splunk/etc/auth/distServerKeys/trusted.pem
06-01-2016 20:50:48.977 +0000 INFO  KeyManagerLocalhost - Finished reading public key for localhost: /opt/splunk/etc/auth/distServerKeys/trusted.pem
06-01-2016 20:50:48.977 +0000 INFO  KeyManagerLocalhost - Reading private key for localhost: /opt/splunk/etc/auth/distServerKeys/private.pem
06-01-2016 20:50:48.977 +0000 INFO  KeyManagerLocalhost - Finished reading private key for localhost: /opt/splunk/etc/auth/distServerKeys/private.pem
06-01-2016 20:50:48.984 +0000 INFO  ServerConfig - Using REMOTE_SERVER_NAME=5453F6EB-0F41-49FA-9203-F6A6FAED2D85
06-01-2016 20:50:48.987 +0000 INFO  ServerRoles - Declared role=search_head.
06-01-2016 20:51:44.987 +0000 INFO  DatabaseDirectoryManager - idx=_internal Writing a bucket manifest in hotWarmPath='/opt/splunk/var/lib/splunk/_internaldb/db', pendingBucketUpdates=0 .  Reason='Buckets were rebuilt or tsidx-minified (bucket_count=1).'
06-01-2016 20:53:43.271 +0000 ERROR SHCRaftConsensus - Failed to bootstrap this node as a captain.
06-01-2016 21:00:23.340 +0000 ERROR SHCRaftConsensus - Failed to bootstrap this node as a captain.
0 Karma

ThomasControlwa
Path Finder

you need min 3 member of SH clustering,
also the member needs to be the same setup. the context of 3 member is to collect a "Captain" role of your shcluster.

make sure your member can connect to the clustermaster and deployer, additional to the deploymentserver

try to fix your quest as following:
setup a new SH member
clean your xx/local/server.conf with all [shcluster] content
only on your SH MEMBER!

-->rm -rf SPLUNK_HOME /etc/instance.cfg
-->rm -rf /SPLUNK_HOME/var/run/splunk/_raft/*
--> ./splunk restart

now build your fresh SH Cluster with
on all sh member
-->./splunk edit cluster-config -master_uri https://xx.xx.xx.xx:8089 -mode slave -site site1 -replication_port 8080 -secret your choice
restart

-->./splunk init shcluster-config -mgmt_uri https://your.sh.member.DNS or IP:8089 -replication_port 8080 -secret changed
-->./splunk bootstrap shcluster-captain -servers_list https://member1:8089,https://member2:8089,https://member3:8089

don't forget to click solfed 🙂

additional links:
http://docs.splunk.com/Documentation/Splunk/7.0.0/DistSearch/SHCconfigurationoverview
hope it helps

dbroggy
Path Finder

I don't understand why Thomas is using the cluster-config command to add a search head. should be the shcluster-config command.

0 Karma

ddrillic
Ultra Champion

@dbroggy, it's an interesting thing, we configure using the cluster-config parameter to set the search head as part of the indexer cluster and then we use the shcluster-config parameter to set it up as part of the search head cluster.

So, the node is part of the indexer cluster and also as part of the sub-cluster of the search head cluster.

0 Karma

dbroggy
Path Finder

But I'm pretty sure that's the wrong syntax.
"-mode slave" assigns an indexer to the indexer cluster.
And search heads don't do replication to an index cluster so replication_port makes no sense.
The command should be:
./splunk edit cluster-config -mode searchhead -site site0 -master_uri https://:8089 -secret pass123
Ref:
https://docs.splunk.com/Documentation/Splunk/6.4.1/DistSearch/SHCandindexercluster

0 Karma

amielke
Communicator

Is the -site option necessary or can I omitting this?

0 Karma

ThomasControlwa
Path Finder

Hi,
I think it depends If your Cluster Is a multisite Cluster, that means you work wit site1 and site2.
If you are not sure, post your SH-Cluster conf
cheers

0 Karma

amielke
Communicator

Oh, we use a single site or say no site Cluster.
Is this in this case necessary?

0 Karma

ThomasControlwa
Path Finder

no, you don't Need this. please take a look @ here http://docs.splunk.com/Documentation/Splunk/7.0.2/Indexer/ConfiguresearchheadwithCLI

Edit the search head settings

You can also use the CLI to edit the configuration later.

Important: When you first enable a search head, you use the splunk edit cluster-config command. To change the search head configuration, you must instead use the splunk edit cluster-master command.

For example, to change the security key (secret), use this command:

splunk edit cluster-master https://10.160.31.200:8089 -secret newsecret123

Important: The splunk edit cluster-master command always takes the current master URI:port value as its initial parameter. For example, this command connects the search head to a different master by setting a new value for the -master_uri parameter, but it provides the value for the old master as its initial parameter:

splunk edit cluster-master https://10.160.31.200:8089 -master_uri https://10.160.31.55:8089

Refer to the CLI clustering help, along with the server.conf specification file, for the list of configurable settings.

0 Karma

ThomasControlwa
Path Finder

please rate my answer 😄 many thanks and cheers

0 Karma

dolivasoh
Contributor

You cannot achieve a quorum with only two search heads. Add another.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...