topic Re: How to back up hot buckets ? in Deployment Architecture

How to back up hot buckets ?

sunrise — Wed, 24 Jul 2013 02:44:19 GMT

Hi Splunkers,

I'm now considering of backing up Splunk indexes to prepare for recovery.
I know that any buckets without hot are able to back up by copy.
Hot buckets are not able to copy because it's always under Splunk process (writing, and so on).

So we have an option of rolling from hot to warm manually,
but it may be not recommended in terms of search performance that we always do that.

And there is an other way to back up hot buckets. That is a snapshot.
According to Splunk documentation, it seems that snapshots enable us to get backups of hot buckets.
However, we don't always have such tools or environments. Splunk clustering either.

Now I want to ask everyone that how to back up hot buckets ?
Giving up to backup hot buckets ?
Thank you for your help.

Re: How to back up hot buckets ?

martin_mueller — Wed, 24 Jul 2013 07:59:45 GMT

I see at least two strategies that work regardless of special tools or snapshot capability.

First, you could schedule a full backup every day or week, and roll hot buckets before that - depending on your specific environment, adding one roll per day or week should not impact search performance too badly. For high-volume indexes you might go from five to six buckets per day, and for low-volume indexes you may go from one bucket per week to one bucket per day - maybe a bit worse if you have several open hot buckets at roll time.

Second, you could rsync (or similar) your warm buckets as soon as splunk rolls them on its own. That way you always loose hot buckets in a crash, but your warm buckets that were backed up may be newer than with method 1. That pretty much depends on how long a bucket remains hot - if that can be weeks then you're better off forcing a roll every now and then.

Re: How to back up hot buckets ?

sunrise — Wed, 24 Jul 2013 08:07:37 GMT

Thank you, martin_mueller. Very good answer !!

Re: How to back up hot buckets ?

anand_singh17 — Wed, 15 Feb 2017 01:17:01 GMT

Hi Martin,

How do you determine the updated bucket across muti-site cluster?

Any link or script to get information ?

Regards,

Re: How to back up hot buckets ?

dwaddle — Wed, 15 Feb 2017 04:02:48 GMT

2017 update. I'm not sure this is even a valid question any more. Every decent Linux distribution -and- modern copy of Windows supports snapshots natively as part of the operating system. In today's world, I can't think of a good reason why you can't use snapshots as your primary backup mechanism, unless you are still using Windows 2003 or Red Hat 5. In which case, upgrading to a modern operating system will give you snapshotting natively.