Deployment Architecture

How to back up hot buckets ?

sunrise
Contributor

Hi Splunkers,

I'm now considering of backing up Splunk indexes to prepare for recovery.
I know that any buckets without hot are able to back up by copy.
Hot buckets are not able to copy because it's always under Splunk process (writing, and so on).

So we have an option of rolling from hot to warm manually,
but it may be not recommended in terms of search performance that we always do that.

And there is an other way to back up hot buckets. That is a snapshot.
According to Splunk documentation, it seems that snapshots enable us to get backups of hot buckets.
However, we don't always have such tools or environments. Splunk clustering either.

Now I want to ask everyone that how to back up hot buckets ?
Giving up to backup hot buckets ?
Thank you for your help.

Tags (2)
1 Solution

martin_mueller
SplunkTrust
SplunkTrust

I see at least two strategies that work regardless of special tools or snapshot capability.

First, you could schedule a full backup every day or week, and roll hot buckets before that - depending on your specific environment, adding one roll per day or week should not impact search performance too badly. For high-volume indexes you might go from five to six buckets per day, and for low-volume indexes you may go from one bucket per week to one bucket per day - maybe a bit worse if you have several open hot buckets at roll time.

Second, you could rsync (or similar) your warm buckets as soon as splunk rolls them on its own. That way you always loose hot buckets in a crash, but your warm buckets that were backed up may be newer than with method 1. That pretty much depends on how long a bucket remains hot - if that can be weeks then you're better off forcing a roll every now and then.

View solution in original post

dwaddle
SplunkTrust
SplunkTrust

2017 update. I'm not sure this is even a valid question any more. Every decent Linux distribution -and- modern copy of Windows supports snapshots natively as part of the operating system. In today's world, I can't think of a good reason why you can't use snapshots as your primary backup mechanism, unless you are still using Windows 2003 or Red Hat 5. In which case, upgrading to a modern operating system will give you snapshotting natively.

martin_mueller
SplunkTrust
SplunkTrust

I see at least two strategies that work regardless of special tools or snapshot capability.

First, you could schedule a full backup every day or week, and roll hot buckets before that - depending on your specific environment, adding one roll per day or week should not impact search performance too badly. For high-volume indexes you might go from five to six buckets per day, and for low-volume indexes you may go from one bucket per week to one bucket per day - maybe a bit worse if you have several open hot buckets at roll time.

Second, you could rsync (or similar) your warm buckets as soon as splunk rolls them on its own. That way you always loose hot buckets in a crash, but your warm buckets that were backed up may be newer than with method 1. That pretty much depends on how long a bucket remains hot - if that can be weeks then you're better off forcing a roll every now and then.

sunrise
Contributor

Thank you, martin_mueller. Very good answer !!

0 Karma

anand_singh17
Path Finder

Hi Martin,

How do you determine the updated bucket across muti-site cluster?

Any link or script to get information ?

Regards,

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...