Why am I getting error "CONFIGURATION ID MISMATCH" trying to add another search head to my search head cluster?

[wweiland]

I'm trying to add another search head to my search head cluster. I'm receiving the following error when I try and bootstrap it.

[labsplunk-sh:/opt/sh2a/bin]$ ./splunk bootstrap shcluster-captain -servers_list "https://#.#.#.#:8088,https://#.#.#.#:8090,https://#.#.#.#:8091,https://#.#.#.#:8092,https://#.#.#.#:8093"

 In handler 'shclustermemberconsensus': CONFIGURATION ID MISMATCH

server.conf
[shclustering]
disabled = 0
mgmt_uri = https://#.#.#.#:8093
pass4SymmKey = XXXXXXXX
adhoc_searchhead = true
conf_deploy_fetch_url = https://#.#.#.#:8088

It's a fresh Splunk tar, but the cluster is in use with deployed apps/configuration, so I don't want to delete everything and start over. The only thing I did to the SH was set the licenser, cluster-config, shcluster-config, and the deployer. This is the only line in the log to indicate the problem.

Anyone else seen this issue or any suggestions on how to fix? I've restarted the whole cluster with no change.

[wweiland · Accepted solution]

Finally got it working. I had to set a static captain, add the system, go back to dynamic captain.

[somesoni2]

Make sure that the conf_deploy_fetch_url is correct and pass4SymmKey is matching with all other existing members.

[somesoni2]

Also, hoping you're following procedure specified here

http://docs.splunk.com/Documentation/Splunk/6.4.2/DistSearch/Addaclustermember#Add_a_new_member