How to ensure no data is lost (add back the databases) if a server is rebuilt using an Ansible script?

[troyfred]

We have an Ansible script that rebuilds/reindexes etc a Splunk indexer, if for some reason it implodes on itself.
We also have incremental backups of the Splunk databases (for this question lets say "Data1").

While the script can rebuild the server, what is the best way to add back those databases if a server is rebuilt so we do not lose all the data we have saved? Thanks in advance for any assistance.

[richgalloway]

The best way is to use an indexer cluster. The cluster automatically backs up buckets to another indexer and automatically recovers from an indexer failure. Your ansible script just has to build the indexer and add it to the cluster then the cluster does the rest.

Your current method most likely is not backing up hot buckets and so you have the risk of losing that data when an indexer fails. Indexer clustering backs up hot buckets continuously so there's little chance of lost data.

[troyfred]

So I would need a second server as basically a back up to the main (or the main being a backup for the backup), so if one fails I blow it away and just have ansible recreate it and the one that didn't die fills in the info?

[richgalloway]

There's a little more to it than that as you'll also need a third server to be the cluster master. Clustered indexers are not active/backup, they are active/active with each taking a portion of the indexing and searching load. See https://docs.splunk.com/Documentation/Splunk/8.0.4/Deploy/Indexercluster