Why am I getting Search Head Cluster Captain bootstrap error "Raft not initialized"?

[chaker]

Hello all,

I get the following error message when bootstraping my Search Head Cluster Captain:

In handler 'shclustermemberconsensus': Raft not initialized. This means that dynamic captain mode was not set in server.conf

./splunk bootstrap shcluster-captain -servers_list "https://x.x.x.x:8089, https://x.x.x.x:8089, https://x.x.x.x:8089, https://x.x.x.x:808"

I have connectivity between SH Heads and Captain, and have followed the documentation. All the searchheads have been initialized correctly and report:
Search head clustering has been initialized on this node.
You need to restart the Splunk Server (splunkd) for your changes to take effect.

Here is the output from a show cluster status on the deployer

root@Deployer:/opt/splunk/bin# ./splunk show shcluster-status
In handler 'shclusterstatus': Search Head Clustering is not enabled on this node. REST endpoint is not available

Dose anyone know why RAFT is not initialized? It's a fresh 6.2 64bit Linux tar of Splunk Enterprise.

Thanks in advance,
Chaker

[chaker · Accepted solution]

I'm glad this was brought up again. I had been incorrectly trying to bootstrap the Deployer as a captain. As arahut_splunk mentions below, the Deployer is not part of the cluster, and should not be a captain. The correct answer to my original question is provided below by arahut_splunk.

The splunk show shcluster-status should not be run on the deployer, it needs to be run on one of the 3 cluster members.
1.think of the deployer as a separate entity outside the shcluster which needs to be able to communicate with the shcluster (using secret /pass4SymmKey)
2.the init step needs to be run on the 3 members. The init step should leave a [shclustering] stanza in your server.conf with correct settings.
3.Bootstrap then needs to be run on only 1 of the members.
4.As mentioned before the status needs to be checked on only the members.

Old Answer
My mistake was not initializing the Deployer with the ./splunk init shcluster-config command. I was trying to ./splunk bootstrap shcluster-captain without the prior command.
I misinterpreted the documentation when reading it.
http://docs.splunk.com/Documentation/Splunk/6.2.1/DistSearch/SHCdeploymentoverview
"This instance cannot be a member of the search head cluster"
I misinterpreted this to mean not to initialize the deployer as a cluster member.

[a212830]

Should this command be run on the deployer? I asked this question recently and was told no, but now I'm having the same error message.

[vnguyen46]

Before bootsrap, don't forget to restart all your splunk shcluster members after bring up and initialize. It worked for me after restarted.

[Steve_G_]

No. You should not run the init shcluster-config command on the deployer. That command is for initializing cluster members only. A deployer cannot be a cluster member.

[a212830]

OK, so how does one address the " 'shclustermemberconsensus': Raft not initialized. This means that dynamic captain mode was not set in server.conf" error message. I don't see that listed in my training notes anywhere.

[arahut_splunk]

The splunk show shcluster-status should not be run on the deployer, it needs to be run on one of the 3 cluster members.

1. think of the deployer as a separate entity outside the shcluster which needs to be able to communicate with the shcluster (using secret /pass4SymmKey)

2. the init step needs to be run on the 3 members. The init step should leave a [shclustering] stanza in your server.conf with correct settings.

3. Bootstrap then needs to be run on only 1 of the members.

4. As mentioned before the status needs to be checked on only the members.

If you still have issues if you follow the steps, please holler.

[esix_splunk]

Have you performed the restart on all of the members and has a captain been succesfully nominated?

./splunk show shcluster-status from the members.