On the clustered indexers, the data you want to keep are :
- the splunk configurations ( SPLUNK_HOME/etc/ usually pushed from your cluster-master), so you could always repush them if you rebuild a new indexer.
- the copies of the buckets in the indexes. Those buckets are replicated across indexers, based on your replication factor RF.
By example with RF =3 , there are 3 copies of each buckets on different peer. So if you lose up to 2 indexers, the data will still be there on the remaining. (who will replicate to meet the RF. You want to bring new indexers to replace them and let the replication occur and optimize the spreading)
If you do not want to just rely on the replication, you can always do additional backups of your indexes folders (by default in SPLUNK_HOME/var/lib/splunk, but you may have custom path, check your indexes.conf homepath and coldpath to figure where)
here is the guide to backup your indexes
https://docs.splunk.com/Documentation/Splunk/8.2.2/Indexer/Backupindexeddata
Remarks :
- doing a backup of live hot buckets (while splunk is on), is not bueno, there is a risk that those files are not in good state when you try to restore them, as they were being written to. You will see in the doc a method to force hot buckets to roll to warm, just before doing a file system backup. (then ignore the new hot buckets during the backup)
- as you are on an indexer cluster, you may have several copies of the same bucket, so your backup will be larger. But there is not easy way to strategize which copy to save to save backup space.
