Deployment Architecture

Why am I getting Search Head Cluster Captain bootstrap error "Raft not initialized"?

Path Finder

Hello all,

I get the following error message when bootstraping my Search Head Cluster Captain:

In handler 'shclustermemberconsensus': Raft not initialized. This means that dynamic captain mode was not set in server.conf

./splunk bootstrap shcluster-captain -servers_list "https://x.x.x.x:8089, https://x.x.x.x:8089, https://x.x.x.x:8089, https://x.x.x.x:808"

I have connectivity between SH Heads and Captain, and have followed the documentation. All the searchheads have been initialized correctly and report:
Search head clustering has been initialized on this node.
You need to restart the Splunk Server (splunkd) for your changes to take effect.

Here is the output from a show cluster status on the deployer

root@Deployer:/opt/splunk/bin# ./splunk show shcluster-status
In handler 'shclusterstatus': Search Head Clustering is not enabled on this node. REST endpoint is not available

Dose anyone know why RAFT is not initialized? It's a fresh 6.2 64bit Linux tar of Splunk Enterprise.

Thanks in advance,
Chaker

1 Solution

Path Finder

I'm glad this was brought up again. I had been incorrectly trying to bootstrap the Deployer as a captain. As arahut_splunk mentions below, the Deployer is not part of the cluster, and should not be a captain. The correct answer to my original question is provided below by arahut_splunk.

The splunk show shcluster-status should not be run on the deployer, it needs to be run on one of the 3 cluster members.
1.think of the deployer as a separate entity outside the shcluster which needs to be able to communicate with the shcluster (using secret /pass4SymmKey)
2.the init step needs to be run on the 3 members. The init step should leave a [shclustering] stanza in your server.conf with correct settings.
3.Bootstrap then needs to be run on only 1 of the members.
4.As mentioned before the status needs to be checked on only the members.

Old Answer
My mistake was not initializing the Deployer with the ./splunk init shcluster-config command. I was trying to ./splunk bootstrap shcluster-captain without the prior command.
I misinterpreted the documentation when reading it.
http://docs.splunk.com/Documentation/Splunk/6.2.1/DistSearch/SHCdeploymentoverview
"This instance cannot be a member of the search head cluster"
I misinterpreted this to mean not to initialize the deployer as a cluster member.

View solution in original post

Path Finder

I'm glad this was brought up again. I had been incorrectly trying to bootstrap the Deployer as a captain. As arahut_splunk mentions below, the Deployer is not part of the cluster, and should not be a captain. The correct answer to my original question is provided below by arahut_splunk.

The splunk show shcluster-status should not be run on the deployer, it needs to be run on one of the 3 cluster members.
1.think of the deployer as a separate entity outside the shcluster which needs to be able to communicate with the shcluster (using secret /pass4SymmKey)
2.the init step needs to be run on the 3 members. The init step should leave a [shclustering] stanza in your server.conf with correct settings.
3.Bootstrap then needs to be run on only 1 of the members.
4.As mentioned before the status needs to be checked on only the members.

Old Answer
My mistake was not initializing the Deployer with the ./splunk init shcluster-config command. I was trying to ./splunk bootstrap shcluster-captain without the prior command.
I misinterpreted the documentation when reading it.
http://docs.splunk.com/Documentation/Splunk/6.2.1/DistSearch/SHCdeploymentoverview
"This instance cannot be a member of the search head cluster"
I misinterpreted this to mean not to initialize the deployer as a cluster member.

View solution in original post

Champion

Should this command be run on the deployer? I asked this question recently and was told no, but now I'm having the same error message.

0 Karma

Contributor

Before bootsrap, don't forget to restart all your splunk shcluster members after bring up and initialize. It worked for me after restarted.

0 Karma

Splunk Employee
Splunk Employee

No. You should not run the init shcluster-config command on the deployer. That command is for initializing cluster members only. A deployer cannot be a cluster member.

Champion

OK, so how does one address the " 'shclustermemberconsensus': Raft not initialized. This means that dynamic captain mode was not set in server.conf" error message. I don't see that listed in my training notes anywhere.

0 Karma

Splunk Employee
Splunk Employee

The splunk show shcluster-status should not be run on the deployer, it needs to be run on one of the 3 cluster members.

  1. think of the deployer as a separate entity outside the shcluster which needs to be able to communicate with the shcluster (using secret /pass4SymmKey)

  2. the init step needs to be run on the 3 members. The init step should leave a [shclustering] stanza in your server.conf with correct settings.

  3. Bootstrap then needs to be run on only 1 of the members.

  4. As mentioned before the status needs to be checked on only the members.

If you still have issues if you follow the steps, please holler.

Splunk Employee
Splunk Employee

Have you performed the restart on all of the members and has a captain been succesfully nominated?

./splunk show shcluster-status from the members.

State of Splunk Careers

Access the Splunk Careers Report to see real data that shows how Splunk mastery increases your value and job satisfaction.

Find out what your skills are worth!