Deployment Architecture

Remove reference to host in mongodb

hettervik
Builder

Hi,

we have a search head cluster where a couple of the search heads where removed by shutting down the VMs. In other words, the search heads wasn't removed gracefully as they should be. Now the remaining search heads is complaining because mongodb can't reach the removed search heads. I'm getting the following error messages:

2017-03-23T12:27:42.296Z I NETWORK  [ReplExecNetThread-1919] getaddrinfo("prod-searchhead-x") failed: Name or service not known

2017-03-23T12:27:42.290Z I REPL     [ReplicationExecutor] Error in heartbeat request to prod-searchhead-x:8191; Location18915 Failed attempt to connect to prod-searchhead-x:8191; couldn't initialize connection to host prod-searchhead-x, address is invalid

Anyone knows how to forefully remove a host from mongodb in the search head cluster, so that we'll get rid of these error messages?

0 Karma
1 Solution

Masa
Splunk Employee
Splunk Employee

As long as current SHC are stable, , in your situation, potentially you can re-build SHC by following the doc below;
http://docs.splunk.com/Documentation/Splunk/6.5.2/DistSearch/Handleraftissues#Fix_the_entire_cluster

If only KVstore is the one complaining and SHC itelf is not looking for the removed SH node anymore, "kvstore resync" will remove the node from the list. Please follow the doc below;
http://docs.splunk.com/Documentation/Splunk/6.5.2/Admin/ResyncKVstore

View solution in original post

0 Karma

Masa
Splunk Employee
Splunk Employee

As long as current SHC are stable, , in your situation, potentially you can re-build SHC by following the doc below;
http://docs.splunk.com/Documentation/Splunk/6.5.2/DistSearch/Handleraftissues#Fix_the_entire_cluster

If only KVstore is the one complaining and SHC itelf is not looking for the removed SH node anymore, "kvstore resync" will remove the node from the list. Please follow the doc below;
http://docs.splunk.com/Documentation/Splunk/6.5.2/Admin/ResyncKVstore

0 Karma

hettervik
Builder

The "splunk resync kvstore" command was just what I was loooking for, thanks! Worked like a charm.

0 Karma

woodcock
Esteemed Legend

Is there some reason, since it is a VM, that you cannot simply restore the machine and follow a proper procedure:
https://docs.splunk.com/Documentation/Splunk/6.5.2/DistSearch/Removeaclustermember

This documentation has the following note:
Important: You must use the procedure documented here to remove a member from the cluster. Do not just stop the member.

0 Karma

hettervik
Builder

Thanks for your comment. We are aware of the procedure in the Splunk documentation, but in this case, what's done is done. We could perhaps set up a new VM, install Splunk on it, and "trick" the search head cluster into believing that this new instance was the previously deleted instance. Then we could properly remove it, though this method takes a lot of extra work. A simpler way would be preferable.

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...