Good morning. I am in a situation where I have no cli (Linux) access to my Index Cluster. I do have the Splunk Secret and have been able to introduce new index peer nodes to the cluster to [hopefully] keep the data. My plan is this:
Let the new indexers sync
Shutdown the inaccessible index peer nodes one site at at time and delete them (This is in AWS). This will hopefully make sure that everything is replicated.
Shutdown the cluster master
Rebuild the master
Reconnect the index peers to the new master. I do plan on changing the pass4SymKey.
The documentation states that I need to backup the server.conf file. Do I really need to do this if I want to rebuild the master? Please share any thoughts/idea that may help me out, I am in a tough spot.
This will work fine as long as you change it on the master first and then the index cluster members. You do need to be aware of what exists on the master-apps folder on the master node. This gets bundled and pushed to all the members. Typically this is the indexers splunk-tcp/ssl inputs and any index time operation knowledge objects.
If you’re good with rest, you should be able to script all this 🙂
THANK YOU so much for this response! I do know the pss4symkey but I would like to change it to be uniform with all the other env's we have. I understand I will need to change it on all nodes (master and peers), should this be an issue? Besides that what other configs do I need? We have a pretty basic index cluster setup.
You can use the transpose command to make these a bit more friendly for recreating config files. But with this, and proper access you should be able to get everything out as long as you know the current pass4symkey.
For the process, you have the general idea correctly. Remove the indexers one peer at a time though, not one site at a time.
** Updated with splunk_server in the rest search. You add the server names to this, otherwise you'll get all configs if youre doing this from the Master Node or MC