Deployment Architecture

Why am I getting Search Head Cluster Captain bootstrap error "Raft not initialized"?

chaker
Contributor

Hello all,

I get the following error message when bootstraping my Search Head Cluster Captain:

In handler 'shclustermemberconsensus': Raft not initialized. This means that dynamic captain mode was not set in server.conf

./splunk bootstrap shcluster-captain -servers_list "https://x.x.x.x:8089, https://x.x.x.x:8089, https://x.x.x.x:8089, https://x.x.x.x:808"

I have connectivity between SH Heads and Captain, and have followed the documentation. All the searchheads have been initialized correctly and report:
Search head clustering has been initialized on this node.
You need to restart the Splunk Server (splunkd) for your changes to take effect.

Here is the output from a show cluster status on the deployer

root@Deployer:/opt/splunk/bin# ./splunk show shcluster-status
In handler 'shclusterstatus': Search Head Clustering is not enabled on this node. REST endpoint is not available

Dose anyone know why RAFT is not initialized? It's a fresh 6.2 64bit Linux tar of Splunk Enterprise.

Thanks in advance,
Chaker

1 Solution

chaker
Contributor

I'm glad this was brought up again. I had been incorrectly trying to bootstrap the Deployer as a captain. As arahut_splunk mentions below, the Deployer is not part of the cluster, and should not be a captain. The correct answer to my original question is provided below by arahut_splunk.

The splunk show shcluster-status should not be run on the deployer, it needs to be run on one of the 3 cluster members.
1.think of the deployer as a separate entity outside the shcluster which needs to be able to communicate with the shcluster (using secret /pass4SymmKey)
2.the init step needs to be run on the 3 members. The init step should leave a [shclustering] stanza in your server.conf with correct settings.
3.Bootstrap then needs to be run on only 1 of the members.
4.As mentioned before the status needs to be checked on only the members.

Old Answer
My mistake was not initializing the Deployer with the ./splunk init shcluster-config command. I was trying to ./splunk bootstrap shcluster-captain without the prior command.
I misinterpreted the documentation when reading it.
http://docs.splunk.com/Documentation/Splunk/6.2.1/DistSearch/SHCdeploymentoverview
"This instance cannot be a member of the search head cluster"
I misinterpreted this to mean not to initialize the deployer as a cluster member.

View solution in original post

chaker
Contributor

I'm glad this was brought up again. I had been incorrectly trying to bootstrap the Deployer as a captain. As arahut_splunk mentions below, the Deployer is not part of the cluster, and should not be a captain. The correct answer to my original question is provided below by arahut_splunk.

The splunk show shcluster-status should not be run on the deployer, it needs to be run on one of the 3 cluster members.
1.think of the deployer as a separate entity outside the shcluster which needs to be able to communicate with the shcluster (using secret /pass4SymmKey)
2.the init step needs to be run on the 3 members. The init step should leave a [shclustering] stanza in your server.conf with correct settings.
3.Bootstrap then needs to be run on only 1 of the members.
4.As mentioned before the status needs to be checked on only the members.

Old Answer
My mistake was not initializing the Deployer with the ./splunk init shcluster-config command. I was trying to ./splunk bootstrap shcluster-captain without the prior command.
I misinterpreted the documentation when reading it.
http://docs.splunk.com/Documentation/Splunk/6.2.1/DistSearch/SHCdeploymentoverview
"This instance cannot be a member of the search head cluster"
I misinterpreted this to mean not to initialize the deployer as a cluster member.

a212830
Champion

Should this command be run on the deployer? I asked this question recently and was told no, but now I'm having the same error message.

0 Karma

vnguyen46
Contributor

Before bootsrap, don't forget to restart all your splunk shcluster members after bring up and initialize. It worked for me after restarted.

0 Karma

Steve_G_
Splunk Employee
Splunk Employee

No. You should not run the init shcluster-config command on the deployer. That command is for initializing cluster members only. A deployer cannot be a cluster member.

a212830
Champion

OK, so how does one address the " 'shclustermemberconsensus': Raft not initialized. This means that dynamic captain mode was not set in server.conf" error message. I don't see that listed in my training notes anywhere.

0 Karma

arahut_splunk
Splunk Employee
Splunk Employee

The splunk show shcluster-status should not be run on the deployer, it needs to be run on one of the 3 cluster members.

  1. think of the deployer as a separate entity outside the shcluster which needs to be able to communicate with the shcluster (using secret /pass4SymmKey)

  2. the init step needs to be run on the 3 members. The init step should leave a [shclustering] stanza in your server.conf with correct settings.

  3. Bootstrap then needs to be run on only 1 of the members.

  4. As mentioned before the status needs to be checked on only the members.

If you still have issues if you follow the steps, please holler.

esix_splunk
Splunk Employee
Splunk Employee

Have you performed the restart on all of the members and has a captain been succesfully nominated?

./splunk show shcluster-status from the members.

Get Updates on the Splunk Community!

How to Monitor Google Kubernetes Engine (GKE)

We’ve looked at how to integrate Kubernetes environments with Splunk Observability Cloud, but what about ...

Index This | How can you make 45 using only 4?

October 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with this ...

Splunk Education Goes to Washington | Splunk GovSummit 2024

If you’re in the Washington, D.C. area, this is your opportunity to take your career and Splunk skills to the ...