All Apps and Add-ons

AWS EBS Snapshots and Splunk indexes

tjj9309
Engager

Is it ok to rely on AWS EBS snapshots to backup splunk data and index? Are there any consistency concerns?

Tags (3)
1 Solution

dolivasoh
Contributor

As far as I know it's pretty decent practice. I also archive buckets to Glacier for good measure.

View solution in original post

HChandler
New Member

I understand that this is an older question, but feel it should be answered anyway.

You can run the snapshots while the SPLUNK systems are running. You can stop the SPLUNK systems to do snapshots as well to make sure all data is copied at that point in time. The only reason to stop SPLUNK to do snapshots, is to collect the "hot" or actively transacted data (actually the hot bucket data cannot be backed up even with a traditional backup program, the data is simply written to the warm bucket when the system is stopped, as warm and cold buckets are the only ones backed up). Most people do not have much data in the hot buckets and are usually not critical to a restoration of the system and data integrity from a restoration point of view. However, if you are paranoid or very critical of your data, then you can stop SPLUNK and do the snapshots. Remember to not restart SPLUNK until all snapshots are done. Otherwise your data may be mismatched as SPLUNK data is spread across the indexers in the system, as no single indexer (unless you only have one indexer) contains all the data in the SPLUNK system. While most other people to save time and resources, running the snapshots while the SPLUNK system is still up is just fine.

0 Karma

HChandler
New Member

I understand that this is an older question, but I feel deserves an answer, regardless of the date of the OP's question.

Doing snapshots of SPLUNK while it still is running should be fine as the only data that would be problematic would be those buckets deemed "hot" (as hot or active transacted data in the hot buckets cannot be backed up even with a true backup program). To have "all" the data you can stop your SPLUNK system until all snapshots are done, as the "hot" data buckets are written to the warm bucket when you stop SPLUNK. Most people do not have much running in the hot bucket at the time and in turn does not become that critical in a restoration of data. But if you are paranoid or super critical of your data, then you can stop the SPLUNK systems until the snapshots are all done. Do not stop one indexer snapshot its volumes and then do the reverse for other indexers in the same system as the data will be miss matched to the point-in-time snapshots.

0 Karma

basharat
New Member

Should I stop Splunk before the periodic snapshots and start again after snapshot complete? What is the best practice to minimize the loss of Splunk Data?

0 Karma

dolivasoh
Contributor

As far as I know it's pretty decent practice. I also archive buckets to Glacier for good measure.

View solution in original post