Monitoring Splunk

Splunk backup and restore procedure?

remy06
Contributor

Hi,

I have attempted to backup and restore splunk by simply tar up the entire splunk directory and restore it to a fresh installation of splunk. However when I start splunk service, splunkd crashes after.

I will require to backup the following:
1) data and indexes
2) configuration settings, users 3) saved searches, reports, field extractions, tags..etc 4) splunk apps

My questions are:
1) Do I need to run the command to roll my hot buckets to warm buckets first?Currently in my "/opt/splunk/var/lib/splunk/defaultdb/db" directory I only see "hot_v1_X"

2) Can I backup/tar the splunk directory without stopping splunk services?

3) In particular,which are the files/directory that I should backup?

Tags (3)

edbolton
Explorer

You don't need to stop Splunk to backup the configs.

As far as the configs go, create a git repo in $SPLUNK_BACKUP (off box, hopefully) then add this cron job however often you want the configs backed up (hourly, daily, decade-ly)

rsync -vaz $SPLUNK_HOME/etc/ $SPLUNK_BACKUP/;
cd $SPLUNK_BACKUP;
git commit -a -m "Configs as of $(date)";

Then you can revert back to whatever version of your configs you want by looking through git log for the date you want to revert to, running git checkout $COMMIT, copying it to your indexer(s), and bouncing splunkd

TonyLeeVT
Builder

Any more details on this? I tried the following which did not work:

cd $SPLUNK_HOME/var/lib/splunk/

tar -zcvf mydatabase.tgz mydatabase

I then copied the .tgz to another splunk instance and untar'd it in the same location. Restarted Splunk and Splunk restarts with the Index disabled. Any ideas?

0 Karma
Get Updates on the Splunk Community!

Splunk Observability Synthetic Monitoring - Resolved Incident on Detector Alerts

We’ve discovered a bug that affected the auto-clear of Synthetic Detectors in the Splunk Synthetic Monitoring ...

Video | Tom’s Smartness Journey Continues

Remember Splunk Community member Tom Kopchak? If you caught the first episode of our Smartness interview ...

3-2-1 Go! How Fast Can You Debug Microservices with Observability Cloud?

3-2-1 Go! How Fast Can You Debug Microservices with Observability Cloud? Learn how unique features like ...