Deployment Architecture

How do you recover an index cluster that we have no access to?

brent_weaver
Builder

Good morning. I am in a situation where I have no cli (Linux) access to my Index Cluster. I do have the Splunk Secret and have been able to introduce new index peer nodes to the cluster to [hopefully] keep the data. My plan is this:

  1. Let the new indexers sync
  2. Shutdown the inaccessible index peer nodes one site at at time and delete them (This is in AWS). This will hopefully make sure that everything is replicated.
  3. Shutdown the cluster master
  4. Rebuild the master
  5. Reconnect the index peers to the new master. I do plan on changing the pass4SymKey.

The documentation states that I need to backup the server.conf file. Do I really need to do this if I want to rebuild the master? Please share any thoughts/idea that may help me out, I am in a tough spot.

Thanks!

0 Karma

esix_splunk
Splunk Employee
Splunk Employee

This will work fine as long as you change it on the master first and then the index cluster members. You do need to be aware of what exists on the master-apps folder on the master node. This gets bundled and pushed to all the members. Typically this is the indexers splunk-tcp/ssl inputs and any index time operation knowledge objects.

If you’re good with rest, you should be able to script all this 🙂

brent_weaver
Builder

THANK YOU so much for this response! I do know the pss4symkey but I would like to change it to be uniform with all the other env's we have. I understand I will need to change it on all nodes (master and peers), should this be an issue? Besides that what other configs do I need? We have a pretty basic index cluster setup.

0 Karma

esix_splunk
Splunk Employee
Splunk Employee

If you have access to the GUI, with admin rights, you can get all the config files and their contents via REST and the configuration endpoints. Start here : http://docs.splunk.com/Documentation/SplunkCloud/6.6.3/RESTREF/RESTconf

Your searches should look something like...

| rest /services/configs/conf-server/clustering splunk_server=local

You can use the transpose command to make these a bit more friendly for recreating config files. But with this, and proper access you should be able to get everything out as long as you know the current pass4symkey.

For the process, you have the general idea correctly. Remove the indexers one peer at a time though, not one site at a time.

** Updated with splunk_server in the rest search. You add the server names to this, otherwise you'll get all configs if youre doing this from the Master Node or MC

Did you miss .conf21 Virtual?

Good news! The event's keynotes and many of its breakout sessions are now available online, and still totally FREE!