Deployment Architecture

Why am I getting Search Head Cluster Captain bootstrap error "Raft not initialized"?

chaker
Contributor

Hello all,

I get the following error message when bootstraping my Search Head Cluster Captain:

In handler 'shclustermemberconsensus': Raft not initialized. This means that dynamic captain mode was not set in server.conf

./splunk bootstrap shcluster-captain -servers_list "https://x.x.x.x:8089, https://x.x.x.x:8089, https://x.x.x.x:8089, https://x.x.x.x:808"

I have connectivity between SH Heads and Captain, and have followed the documentation. All the searchheads have been initialized correctly and report:
Search head clustering has been initialized on this node.
You need to restart the Splunk Server (splunkd) for your changes to take effect.

Here is the output from a show cluster status on the deployer

root@Deployer:/opt/splunk/bin# ./splunk show shcluster-status
In handler 'shclusterstatus': Search Head Clustering is not enabled on this node. REST endpoint is not available

Dose anyone know why RAFT is not initialized? It's a fresh 6.2 64bit Linux tar of Splunk Enterprise.

Thanks in advance,
Chaker

1 Solution

chaker
Contributor

I'm glad this was brought up again. I had been incorrectly trying to bootstrap the Deployer as a captain. As arahut_splunk mentions below, the Deployer is not part of the cluster, and should not be a captain. The correct answer to my original question is provided below by arahut_splunk.

The splunk show shcluster-status should not be run on the deployer, it needs to be run on one of the 3 cluster members.
1.think of the deployer as a separate entity outside the shcluster which needs to be able to communicate with the shcluster (using secret /pass4SymmKey)
2.the init step needs to be run on the 3 members. The init step should leave a [shclustering] stanza in your server.conf with correct settings.
3.Bootstrap then needs to be run on only 1 of the members.
4.As mentioned before the status needs to be checked on only the members.

Old Answer
My mistake was not initializing the Deployer with the ./splunk init shcluster-config command. I was trying to ./splunk bootstrap shcluster-captain without the prior command.
I misinterpreted the documentation when reading it.
http://docs.splunk.com/Documentation/Splunk/6.2.1/DistSearch/SHCdeploymentoverview
"This instance cannot be a member of the search head cluster"
I misinterpreted this to mean not to initialize the deployer as a cluster member.

View solution in original post

chaker
Contributor

I'm glad this was brought up again. I had been incorrectly trying to bootstrap the Deployer as a captain. As arahut_splunk mentions below, the Deployer is not part of the cluster, and should not be a captain. The correct answer to my original question is provided below by arahut_splunk.

The splunk show shcluster-status should not be run on the deployer, it needs to be run on one of the 3 cluster members.
1.think of the deployer as a separate entity outside the shcluster which needs to be able to communicate with the shcluster (using secret /pass4SymmKey)
2.the init step needs to be run on the 3 members. The init step should leave a [shclustering] stanza in your server.conf with correct settings.
3.Bootstrap then needs to be run on only 1 of the members.
4.As mentioned before the status needs to be checked on only the members.

Old Answer
My mistake was not initializing the Deployer with the ./splunk init shcluster-config command. I was trying to ./splunk bootstrap shcluster-captain without the prior command.
I misinterpreted the documentation when reading it.
http://docs.splunk.com/Documentation/Splunk/6.2.1/DistSearch/SHCdeploymentoverview
"This instance cannot be a member of the search head cluster"
I misinterpreted this to mean not to initialize the deployer as a cluster member.

a212830
Champion

Should this command be run on the deployer? I asked this question recently and was told no, but now I'm having the same error message.

0 Karma

vnguyen46
Contributor

Before bootsrap, don't forget to restart all your splunk shcluster members after bring up and initialize. It worked for me after restarted.

0 Karma

Steve_G_
Splunk Employee
Splunk Employee

No. You should not run the init shcluster-config command on the deployer. That command is for initializing cluster members only. A deployer cannot be a cluster member.

a212830
Champion

OK, so how does one address the " 'shclustermemberconsensus': Raft not initialized. This means that dynamic captain mode was not set in server.conf" error message. I don't see that listed in my training notes anywhere.

0 Karma

arahut_splunk
Splunk Employee
Splunk Employee

The splunk show shcluster-status should not be run on the deployer, it needs to be run on one of the 3 cluster members.

  1. think of the deployer as a separate entity outside the shcluster which needs to be able to communicate with the shcluster (using secret /pass4SymmKey)

  2. the init step needs to be run on the 3 members. The init step should leave a [shclustering] stanza in your server.conf with correct settings.

  3. Bootstrap then needs to be run on only 1 of the members.

  4. As mentioned before the status needs to be checked on only the members.

If you still have issues if you follow the steps, please holler.

esix_splunk
Splunk Employee
Splunk Employee

Have you performed the restart on all of the members and has a captain been succesfully nominated?

./splunk show shcluster-status from the members.

Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...