Deployment Architecture

After removing a temporary search head without removing it first as a search head cluster member, how to troubleshoot error "CONFIGURATION ID MISMATCH"?

Motivator

Hey Experts,

I have a cluster setup with 3 search heads and 1 deployer. Out of the three search heads, one was temporary. Once i got the permanent, i being stupid and without thinking, removed the temporary without removing it being part of the cluster members. I decided to redo the whole deployer setup. Got deployer set, initialized cluster members done. Can't elect captain as i get this message:

In handler 'shclustermemberconsensus': CONFIGURATION ID MISMATCH

When i looked in the splunkd.log, it was looking for the temporary cluster member which was a part of it initially. Now it's gone. I just can't elect a captain. Is there any other way I can convince the SH cluster member I want to be captain that the temporary SH I added earlier is no more and R.I.P?

Thanks,
Raghav

1 Solution

Motivator

I had to do a ./splunk stop on all instances, clean and re-configure. Glad those were brand new sets. Just an FYI, if you want you remove/replace an existing cluster member, please follow the documentation.

http://docs.splunk.com/Documentation/Splunk/6.2.1/DistSearch/Removeaclustermember

splunk remove shcluster-member -mgmt_uri <URI>:<management_port>

Thanks,
Raghav

View solution in original post

Motivator

I had to do a ./splunk stop on all instances, clean and re-configure. Glad those were brand new sets. Just an FYI, if you want you remove/replace an existing cluster member, please follow the documentation.

http://docs.splunk.com/Documentation/Splunk/6.2.1/DistSearch/Removeaclustermember

splunk remove shcluster-member -mgmt_uri <URI>:<management_port>

Thanks,
Raghav

View solution in original post

Motivator

Minimum members for a shc is 3. A captain normally won't be elected with only 2 members as there isn't a deciding vote.

0 Karma
State of Splunk Careers

Access the Splunk Careers Report to see real data that shows how Splunk mastery increases your value and job satisfaction.

Find out what your skills are worth!