Getting Data In

How to ensure no data is lost (add back the databases) if a server is rebuilt using an Ansible script?

troyfred
Explorer

We have an Ansible script that rebuilds/reindexes etc a Splunk indexer, if for some reason it implodes on itself.
We also have incremental backups of the Splunk databases (for this question lets say "Data1").

While the script can rebuild the server, what is the best way to add back those databases if a server is rebuilt so we do not lose all the data we have saved? Thanks in advance for any assistance.

Labels (1)
0 Karma

richgalloway
SplunkTrust
SplunkTrust

The best way is to use an indexer cluster. The cluster automatically backs up buckets to another indexer and automatically recovers from an indexer failure. Your ansible script just has to build the indexer and add it to the cluster then the cluster does the rest.

Your current method most likely is not backing up hot buckets and so you have the risk of losing that data when an indexer fails. Indexer clustering backs up hot buckets continuously so there's little chance of lost data.

---
If this reply helps you, Karma would be appreciated.
0 Karma

troyfred
Explorer

So I would need a second server as basically a back up to the main (or the main being a backup for the backup), so if one fails I blow it away and just have ansible recreate it and the one that didn't die fills in the info?

0 Karma

richgalloway
SplunkTrust
SplunkTrust

There's a little more to it than that as you'll also need a third server to be the cluster master. Clustered indexers are not active/backup, they are active/active with each taking a portion of the indexing and searching load. See https://docs.splunk.com/Documentation/Splunk/8.0.4/Deploy/Indexercluster

---
If this reply helps you, Karma would be appreciated.
0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...