How to setup search head clustering?

[damonmanni]

I've done all the steps (several times) in the docs to setup a SH cludster from scratch configureing deployer; initialize each SH member, and then set a captain. But each time I run the cmd to set the captain (splunk bootstrap shcluster-captain ....) the cmd will just hang forever, timeout, or throw the error "In handler 'shclustermemberconsensus': CONFIGURATION ID MISMATCH". But then if I run the cmd (splunk show shcluster-status ) it does show a captain set and members. In other words it does provide a result which looks correct to me. How can there be a captain set?

My question is "how do I really know if the captain has been set since I never got the bootstrap cmd to get a good result?

I tried many approaches to get the cmd to run but no luck on any of my attempts:

• Changed the pass4symkey on BOTH the [general] & [shclustering] stanzas so they match across the deployer, and the 2 SH members + restarts each time.

• I also tried the same text string and 2 different text strings with no special characters between the two stanzas. bootstrap cmd still fails..

• I did notice that the hashed result string on the 2 SH members always match, but the hash on the deployer does not match the other 2 members (even though I started off with the same text string to be encrypted. Don't know if this really means anything but to have a different hash value between the deployer & members seems normal.

• I did full tear down's and rebuilds of deployer, and 2 SH members + all the steps again. bootstrap cmd still fails..

• I even hit the individual mgmt_uri's (mgmt_uri = https://mdcsueve.fer.com:8089) in a browser and they all came back with data. bootstrap cmd still fails..

• These are brand new vanilla VM's working fine. I installed fresh splunk copies v6.4.1 with no issues.

• I tried different replication ports also just to make sure they were not being used. Still bad results.

I am out of ideas, but need the SH cluster setup. Any help is appreciated.

DOC used: https://docs.splunk.com/Documentation/Splunk/6.4.1/DistSearch/SHCdeploymentoverview

[richgalloway · Accepted solution]

The splunk show shcluster-status command has confirmed you have a captain set. Continue from there.

[damonmanni]

Yes, as mentioned I ran this command already. The problem is the bootstrap cmd (the step before the status cmd) never completed as I mentioned, so how can the status cmd show I have a captain. doesn't make sense.