After removing a temporary search head without removing it first as a search head cluster member, how to troubleshoot error "CONFIGURATION ID MISMATCH"?

[Raghav2384]

Hey Experts,

I have a cluster setup with 3 search heads and 1 deployer. Out of the three search heads, one was temporary. Once i got the permanent, i being stupid and without thinking, removed the temporary without removing it being part of the cluster members. I decided to redo the whole deployer setup. Got deployer set, initialized cluster members done. Can't elect captain as i get this message:

In handler 'shclustermemberconsensus': CONFIGURATION ID MISMATCH

When i looked in the splunkd.log, it was looking for the temporary cluster member which was a part of it initially. Now it's gone. I just can't elect a captain. Is there any other way I can convince the SH cluster member I want to be captain that the temporary SH I added earlier is no more and R.I.P?

Thanks,
Raghav

[Raghav2384 · Accepted solution]

I had to do a ./splunk stop on all instances, clean and re-configure. Glad those were brand new sets. Just an FYI, if you want you remove/replace an existing cluster member, please follow the documentation.

http://docs.splunk.com/Documentation/Splunk/6.2.1/DistSearch/Removeaclustermember

splunk remove shcluster-member -mgmt_uri <URI>:<management_port>

Thanks,
Raghav

[Lucas_K]

Minimum members for a shc is 3. A captain normally won't be elected with only 2 members as there isn't a deciding vote.