Search Head Clustering fails to work..

[dilipbailwal]

As per the documents, i have successfully configured deployer and then further initialized all the Search Members too.
Now, while Bringing up Captain continuously failed .. Not sure why we have to define captain if the Election is Dynamic.

Anyways, here is the error what i am getting after executing following command,
./splunk show shcluster-status -auth admin username:password

> This node is not the captain of the
> search head pool, and we could not
> determine the current captain. The
> pool is either in the process of
> electing a new captain, or this member
> hasn't joined the pool.

after a while i checked and still captain wasn't elected.
I also ran bootstrap command to bring up captain and it failed. with error,

> In handler
> 'shclustermemberconsensus': Failed to
> bootstrap this node as a captain.

and then one of the node where bootstrap produces result,

> In handler 'shclustermemberconsensus':
> NOT_LEADER CURRENT_STATE = FOLLOWER

[dilipbailwal · Accepted solution]

Issue is resolved.

The problem was, one of the servers in SH cluster had a wrong mgmt_uri.. hence it wasn't connecting..
After correcting the server, it worked and also i did bootstrap on another server as it has to be executed only once on a particular server.

Thanks amiracle for your reply

[amiracle]

Did you try modifying the server.conf and make sure that your mgmt_uri is the same as the search head that you are trying to start?

Here is a sample entry in my server.conf (assuming my search head cluster member is splunk05 and my deployer is splunk01):

[general]
pass4SymmKey = <some_passkey>
serverName = splunk05

[replication_port://34567]

[shclustering]
conf_deploy_fetch_url = https://splunk01:8089
disabled = 0
mgmt_uri = https://splunk05:8089
pass4SymmKey = <some_encrypted_passkey>
replication_factor = 3
id = <some_generated_id>

I had the same problem and it turns out my mgmt_uri was pointing to the previous captain, something I think I did by accident during my adding of the cluster member.

[hitesh_kanchan]

Did you try running "./splunk bootstrap" in another Search head member as well.

If that doesn't help, try setting pass4Symmkey=someValue in [shclustering] in server.conf and perform "./splunk init with -secret someValue" in the Search head members.