Deployment Architecture

After removing a temporary search head without removing it first as a search head cluster member, how to troubleshoot error "CONFIGURATION ID MISMATCH"?

Raghav2384
Motivator

Hey Experts,

I have a cluster setup with 3 search heads and 1 deployer. Out of the three search heads, one was temporary. Once i got the permanent, i being stupid and without thinking, removed the temporary without removing it being part of the cluster members. I decided to redo the whole deployer setup. Got deployer set, initialized cluster members done. Can't elect captain as i get this message:

In handler 'shclustermemberconsensus': CONFIGURATION ID MISMATCH

When i looked in the splunkd.log, it was looking for the temporary cluster member which was a part of it initially. Now it's gone. I just can't elect a captain. Is there any other way I can convince the SH cluster member I want to be captain that the temporary SH I added earlier is no more and R.I.P?

Thanks,
Raghav

1 Solution

Raghav2384
Motivator

I had to do a ./splunk stop on all instances, clean and re-configure. Glad those were brand new sets. Just an FYI, if you want you remove/replace an existing cluster member, please follow the documentation.

http://docs.splunk.com/Documentation/Splunk/6.2.1/DistSearch/Removeaclustermember

splunk remove shcluster-member -mgmt_uri <URI>:<management_port>

Thanks,
Raghav

View solution in original post

Raghav2384
Motivator

I had to do a ./splunk stop on all instances, clean and re-configure. Glad those were brand new sets. Just an FYI, if you want you remove/replace an existing cluster member, please follow the documentation.

http://docs.splunk.com/Documentation/Splunk/6.2.1/DistSearch/Removeaclustermember

splunk remove shcluster-member -mgmt_uri <URI>:<management_port>

Thanks,
Raghav

View solution in original post

Lucas_K
Motivator

Minimum members for a shc is 3. A captain normally won't be elected with only 2 members as there isn't a deciding vote.

0 Karma
Register for .conf21 Now! Go Vegas or Go Virtual!

How will you .conf21? You decide! Go in-person in Las Vegas, 10/18-10/21, or go online with .conf21 Virtual, 10/19-10/20.