Deployment Architecture

How to setup search head clustering?

damonmanni
Path Finder

I've done all the steps (several times) in the docs to setup a SH cludster from scratch configureing deployer; initialize each SH member, and then set a captain. But each time I run the cmd to set the captain (splunk bootstrap shcluster-captain ....) the cmd will just hang forever, timeout, or throw the error "In handler 'shclustermemberconsensus': CONFIGURATION ID MISMATCH". But then if I run the cmd (splunk show shcluster-status ) it does show a captain set and members. In other words it does provide a result which looks correct to me. How can there be a captain set?

My question is "how do I really know if the captain has been set since I never got the bootstrap cmd to get a good result?

I tried many approaches to get the cmd to run but no luck on any of my attempts:

  • Changed the pass4symkey on BOTH the [general] & [shclustering] stanzas so they match across the deployer, and the 2 SH members + restarts each time.

  • I also tried the same text string and 2 different text strings with no special characters between the two stanzas. bootstrap cmd still fails..

  • I did notice that the hashed result string on the 2 SH members always match, but the hash on the deployer does not match the other 2 members (even though I started off with the same text string to be encrypted. Don't know if this really means anything but to have a different hash value between the deployer & members seems normal.

  • I did full tear down's and rebuilds of deployer, and 2 SH members + all the steps again. bootstrap cmd still fails..

  • I even hit the individual mgmt_uri's (mgmt_uri = https://mdcsueve.fer.com:8089) in a browser and they all came back with data. bootstrap cmd still fails..

  • These are brand new vanilla VM's working fine. I installed fresh splunk copies v6.4.1 with no issues.

  • I tried different replication ports also just to make sure they were not being used. Still bad results.

I am out of ideas, but need the SH cluster setup. Any help is appreciated.

DOC used: https://docs.splunk.com/Documentation/Splunk/6.4.1/DistSearch/SHCdeploymentoverview

0 Karma
1 Solution

richgalloway
SplunkTrust
SplunkTrust

The splunk show shcluster-status command has confirmed you have a captain set. Continue from there.

---
If this reply helps you, an upvote would be appreciated.

View solution in original post

0 Karma

richgalloway
SplunkTrust
SplunkTrust

The splunk show shcluster-status command has confirmed you have a captain set. Continue from there.

---
If this reply helps you, an upvote would be appreciated.

View solution in original post

0 Karma

damonmanni
Path Finder

Yes, as mentioned I ran this command already. The problem is the bootstrap cmd (the step before the status cmd) never completed as I mentioned, so how can the status cmd show I have a captain. doesn't make sense.

0 Karma
.conf21 CFS Extended through 5/20!

Don't miss your chance
to share your Splunk
wisdom in-person or
virtually at .conf21!

Call for Speakers has
been extended through
Thursday, 5/20!