Does Splunk automatically rotate all buckets being restored when doing a backup?

[Arkon]

Hello,

I am currently designing a backup/restore procedure and I am wondering:
Does Splunk automatically rotate all the buckets being restored?

I am wondering if I should manually rotate the buckets when I do my incremental backup, or if I should save them all and let Splunk rotate them when I restore them?

If Splunk is able to load, in thawdb, let's say, 600 warm buckets for a limit of 300 and rotate to cold the corresponding ones and then move the cold ones to frozen, then all I will have to do, for backup, is:
- Upload recently warmed up ones
- Delete backup of buckets older than frozen time from long-term storage

If not, I will keep a directory for each warm and cold buckets and do the upload/rotation/deletion at every incremental backup round. On the restore, I will restore them by age from oldest to most recent, which is much more painful.

Thanks a lot

[harsmarvania57 · Accepted solution]

Hi @Arkon,

As per splunk documentation http://docs.splunk.com/Documentation/Splunk/6.4.0/Indexer/Backupindexeddata , if you will take backup of hot buckets then it will not useful when you will restore them. So you can take backup of Warm and Cold buckets.

To summarize:

 hot buckets - Currently being written to; do not back these up.
 warm buckets - Rolled from hot; can be safely backed up.
 cold buckets - Rolled from warm; buckets are moved to another location.
 frozen buckets - The indexer deletes these, but you can archive their contents first.

1. If you will manually rotate buckets when you will take incremental backup then there will be chances in which your environment will be running with too many buckets.
2. If you will restore buckets in thawdb then splunk will not delete those buckets after it reaches their retention period you need to manually delete those buckets from thawdb folders.

[harsmarvania57]

If you are running Indexer Cluster and if you will have too many buckets then you might face issue which is mentioned in https://answers.splunk.com/answers/233441/cluster-master-is-unable-to-meet-search-factor-and.html

1. When you restore Warm Bucket or Cold Bucket to thawdb, you just need to restore db folder starting with db_xxxxxx_xxxxxx and it will stay in thawdb only it will not create any new bucket. (Something like copy and paste from backup server to splunk server)
2. You need to remove bucket manually from thawdb , splunk will not remove those buckets which are placed in thawdb.
3. For suggestion there are many things which need to consider, 1.) what is your bucket side (maxDataSize in indexes.conf), if bucket size is very small and you will run backup daily then there will be more buckets in your environment. 2.) How many indexes you have, because each index have their own buckets, if you have many indexes (In hundreds or thousands) and you want to take backup daily. I'll not suggest to role hot bucket to warm for backup purpose. 3.) How many buckets generating each day in your indexes. 4.) Backup frequency. These points you need to consider if you want to roll hot bucket manually to warm when taking backup.

Hope this clarifies your query.

Thanks,
Harshil

[Arkon]

Thanks a lot!
Just one final question:
What the difference between restoring in thaweddb beween restoring in db/ please?

[harsmarvania57]

If you restore in thaweddb then you can use those buckets directly without restarting splunk but it will not remove when it will age.
If you restore in db folder then you need to reindex whole index but I have never tried to restore in db folder.

[Arkon]

Thanks!
What are the consequences of running with too many buckets?

When I restore them in thawdb, is the data copied from thawdb to a new bucket or is it really staying in this directory?
If it stays and doesn't migrate, thawdb can become very big at a point.

Do you finally suggest to do the rotation manually when backuping so, when restoring, there is already the right amount of bucket?