Deployment Architecture

After removing a temporary search head without removing it first as a search head cluster member, how to troubleshoot error "CONFIGURATION ID MISMATCH"?

Raghav2384
Motivator

Hey Experts,

I have a cluster setup with 3 search heads and 1 deployer. Out of the three search heads, one was temporary. Once i got the permanent, i being stupid and without thinking, removed the temporary without removing it being part of the cluster members. I decided to redo the whole deployer setup. Got deployer set, initialized cluster members done. Can't elect captain as i get this message:

In handler 'shclustermemberconsensus': CONFIGURATION ID MISMATCH

When i looked in the splunkd.log, it was looking for the temporary cluster member which was a part of it initially. Now it's gone. I just can't elect a captain. Is there any other way I can convince the SH cluster member I want to be captain that the temporary SH I added earlier is no more and R.I.P?

Thanks,
Raghav

1 Solution

Raghav2384
Motivator

I had to do a ./splunk stop on all instances, clean and re-configure. Glad those were brand new sets. Just an FYI, if you want you remove/replace an existing cluster member, please follow the documentation.

http://docs.splunk.com/Documentation/Splunk/6.2.1/DistSearch/Removeaclustermember

splunk remove shcluster-member -mgmt_uri <URI>:<management_port>

Thanks,
Raghav

View solution in original post

Raghav2384
Motivator

I had to do a ./splunk stop on all instances, clean and re-configure. Glad those were brand new sets. Just an FYI, if you want you remove/replace an existing cluster member, please follow the documentation.

http://docs.splunk.com/Documentation/Splunk/6.2.1/DistSearch/Removeaclustermember

splunk remove shcluster-member -mgmt_uri <URI>:<management_port>

Thanks,
Raghav

Lucas_K
Motivator

Minimum members for a shc is 3. A captain normally won't be elected with only 2 members as there isn't a deciding vote.

0 Karma
Get Updates on the Splunk Community!

How I instrumented a Rust application without knowing Rust

As a technical writer, I often have to edit or create code snippets for Splunk's distributions of ...

Splunk Community Platform Survey

Hey Splunk Community, Starting today, the community platform may prompt you to participate in a survey. The ...

Observability Highlights | November 2022 Newsletter

 November 2022Observability CloudEnd Of Support Extension for SignalFx Smart AgentSplunk is extending the End ...