topic Re: Data backup in Deployment Architecture

Data backup

tekebede — Thu, 13 Aug 2020 21:43:18 GMT

Hello,

what is the recommended way to automate a db backup to QNAP NAS every 24 hrs. (SPLUNK enterprise 8.0.4). is there a way to do it from the GUI or it must be done at the CLI. The server is running low on disk space and i have a 24 TB qnap sitting around.

Re: Data backup

MaverickT — Fri, 14 Aug 2020 09:05:28 GMT

There is no way of doing Splunk database backup via GUI. Even the CLI might get a little bit complicated, because you have to roll the buckets before making backup. You can learn more about making backup here:

https://docs.splunk.com/Documentation/Splunk/8.0.5/Indexer/Backupindexeddata

https://docs.splunk.com/Documentation/Splunk/8.0.5/Admin/BackupKVstore

Maybe you should consider moving the cold buckets to the QNAP NAS?

Re: Data backup

tekebede — Fri, 14 Aug 2020 14:49:06 GMT

Thanks, MaverickT

I was thinking if I reduce the retention policy on the tsidx files the aged files will roll to cold bucket and i can move that to the NAS. but how do I move the cold buckets to the NAS server? both servers are on the same vlan.

Re: Data backup

isoutamo — Fri, 14 Aug 2020 15:57:55 GMT

It depends what OS you are using for splunk (Linux, Windows) and is it distributed or only all in one solution?

Anyhow 1st thing is to ensure that QNAS has enough IOPS for splunk (800 is minimum). If not then you should forget it. If it has and you can trust that it servers your otherwise enough well then you could go forward.

Probably best (or not worst) option is define iSCSI LUNs on QNAP and then present those to your splunk indexers (use at least two network path). Add those to own "volume group" and add splunk volume over it. Then just configure splunk's indexes to use it. Then use that volume as cold storege for your data.

r. Ismo

Re: Data backup

tekebede — Mon, 17 Aug 2020 14:24:05 GMT

Mr. Soutamo

Sorry for the delay but i am waiting for the network guys to drop a cable so i can make a SCASI connection and configure the QNAP. but what i didn't understand is when you say " present those to your splunk indexers use at least two network path. (Add those to own "volume group" and add splunk volume over it). Then just configure splunk's indexes to use it. how do i present that to SPLUNK? as you can see i am a rookie on both splunk and qnap

Re: Data backup

rabbidroid — Mon, 17 Aug 2020 17:11:43 GMT

You can only back up warm or cold buckets. Here is a quick and dirty way to do it:

If you are running Linux, this will copy all non-hot buckets created in the last day to /tmp. replace /tmp with you desired target directory

warm_buckets=$(find /opt/splunk/var/lib/splunk -mmin -1440 -type d -name "db_*")

for i in $warm_buckets; do mkdir -p /tmp/$i/rawdata; done

for i in $warm_buckets; do rsync -auvn "$i"/*/journal.gz /tmp$i; done

Hope this helps

Re: Data backup

tekebede — Mon, 17 Aug 2020 20:40:51 GMT

I have successfully completed configuring the scsi connection and it shows as a disk on the server where spunk resides on. i managed to move about 200 GB of files from the warm bucket temporarily (manually) but i want to automate the process. I will do that on the cold bucket once i figure out how to increase the rolling time from the worm bucket to the cold and also roll out time from the hot to worm.

Grateful for your help

Tess