I have attempted to backup and restore splunk by simply tar up the entire splunk directory and restore it to a fresh installation of splunk. However when I start splunk service, splunkd crashes after.
I will require to backup the following:
1) data and indexes
2) configuration settings, users 3) saved searches, reports, field extractions, tags..etc 4) splunk apps
My questions are:
1) Do I need to run the command to roll my hot buckets to warm buckets first?Currently in my "/opt/splunk/var/lib/splunk/defaultdb/db" directory I only see "hot_v1_X"
2) Can I backup/tar the splunk directory without stopping splunk services?
3) In particular,which are the files/directory that I should backup?
You don't need to stop Splunk to backup the configs.
As far as the configs go, create a git repo in $SPLUNK_BACKUP (off box, hopefully) then add this cron job however often you want the configs backed up (hourly, daily, decade-ly)
rsync -vaz $SPLUNK_HOME/etc/ $SPLUNK_BACKUP/; cd $SPLUNK_BACKUP; git commit -a -m "Configs as of $(date)";
Then you can revert back to whatever version of your configs you want by looking through
git log for the date you want to revert to, running
git checkout $COMMIT, copying it to your indexer(s), and bouncing splunkd
Any more details on this? I tried the following which did not work:
cd $SPLUNK_HOME/var/lib/splunk/ tar -zcvf mydatabase.tgz mydatabase
I then copied the .tgz to another splunk instance and untar'd it in the same location. Restarted Splunk and Splunk restarts with the Index disabled. Any ideas?