What is the best practice to backup Splunk?

[mlevsh]

Hi,
We have Splunk v. 6.3.3 multi site indexer cluster (8 indexers), 4 search heads, deployment server.
What's the best approach to backup policy?
1. Should we backup $SPLUNK_HOME/etc directory on servers?
2. Will backing up of diag file be useful?
3. Any other advices?

Thank you in advance!

[maraman_splunk]

Hi,

for configuration, state and kvstore backup, you could use splunkconf-backup app (https://splunkbase.splunk.com/app/5600 , https://github.com/splunk/splunkconf-backup/wiki )

for indexed data,

• in classic architecture, you mainly have to rely on traditional backup solutions (and their challenges)
• in Smartstore architecture , you could leverage object store feature like versioning, lifecycle policy for delete, compliance option like objectlock and/or ability to sync object to another site in the background, depending on what you want to achieve

[skoelpin]

The main thing you want to backup is your hot/warm/cold/frozen db's which are under SPLUNK_HOME/var/lib/splunk/defaultdb/db and SPLUNK_HOME/var/lib/splunk/defaultdb/colddb along with your configs under SPLUNK_HOME/etc/system/local

Don't worry about backing up the diag's since it's just diagnostics on the current state of your system

How do you plan on backing this up? Are you running on a SAN? If so, do you plan on taking daily snapshots?

http://docs.splunk.com/Documentation/Splunk/6.2.0/Indexer/HowSplunkstoresindexes

[ehudb · Accepted solution]

1.. Backup:

Search heads:

If the search heads are in a cluster,
- backup the deployer $SPLUNK_HOME\etc\shcluster
- $SPLUNK_HOME\etc of one the peers

Otherwise:
- backup $SPLUNK_HOME\etc of all the search heads

Indexers:
Backup warm\cold\frozen buckets
Also backup cluster master - master apps.

Deployment (Forwarder deployment?):
Backup $SPLUNK_HOME\etc directory

2..
I wouldn't backup diag file as it contains more than the needed files

3..
Docs about backup:
Indexed data (bcukets) - http://docs.splunk.com/Documentation/Splunk/6.5.0/Indexer/Backupindexeddata
Configuration (etc) - http://docs.splunk.com/Documentation/Splunk/6.5.0/Admin/Backupconfigurations

[SamHTexas]

Reg post below by ehudb. What else should I be backing up & how often? Is backing up the Index data really necessary? 

1.. Backup:

Search heads:

If the search heads are in a cluster,
- backup the deployer $SPLUNK_HOME\etc\shcluster
- $SPLUNK_HOME\etc of one the peers

Otherwise:
- backup $SPLUNK_HOME\etc of all the search heads

Indexers:
Backup warm\cold\frozen buckets
Also backup cluster master - master apps.

Deployment (Forwarder deployment?):
Backup $SPLUNK_HOME\etc directory

2..
I wouldn't backup diag file as it contains more than the needed files

3..
Docs about backup:
Indexed data (bcukets) - http://docs.splunk.com/Documentation/Splunk/6.5.0/Indexer/Backupindexeddata
Configuration (etc) - http://docs.splunk.com/Documentation/Splunk/6.5.0/Admin/Backupconfigurations

[mlevsh]

@ehudb thank you!