I've done all the steps (several times) in the docs to setup a SH cludster from scratch configureing deployer; initialize each SH member, and then set a captain. But each time I run the cmd to set the captain (splunk bootstrap shcluster-captain ....) the cmd will just hang forever, timeout, or throw the error "In handler 'shclustermemberconsensus': CONFIGURATION ID MISMATCH". But then if I run the cmd (splunk show shcluster-status ) it does show a captain set and members. In other words it does provide a result which looks correct to me. How can there be a captain set?
My question is "how do I really know if the captain has been set since I never got the bootstrap cmd to get a good result?
I tried many approaches to get the cmd to run but no luck on any of my attempts:
Changed the pass4symkey on BOTH the [general] & [shclustering] stanzas so they match across the deployer, and the 2 SH members + restarts each time.
I also tried the same text string and 2 different text strings with no special characters between the two stanzas. bootstrap cmd still fails..
I did notice that the hashed result string on the 2 SH members always match, but the hash on the deployer does not match the other 2 members (even though I started off with the same text string to be encrypted. Don't know if this really means anything but to have a different hash value between the deployer & members seems normal.
I did full tear down's and rebuilds of deployer, and 2 SH members + all the steps again. bootstrap cmd still fails..
I even hit the individual mgmt_uri's (mgmt_uri = https://mdcsueve.fer.com:8089) in a browser and they all came back with data. bootstrap cmd still fails..
These are brand new vanilla VM's working fine. I installed fresh splunk copies v6.4.1 with no issues.
I tried different replication ports also just to make sure they were not being used. Still bad results.
I am out of ideas, but need the SH cluster setup. Any help is appreciated.
DOC used: https://docs.splunk.com/Documentation/Splunk/6.4.1/DistSearch/SHCdeploymentoverview
