I am currently designing a backup/restore procedure and I am wondering:
Does Splunk automatically rotate all the buckets being restored?
I am wondering if I should manually rotate the buckets when I do my incremental backup, or if I should save them all and let Splunk rotate them when I restore them?
If Splunk is able to load, in thawdb, let's say, 600 warm buckets for a limit of 300 and rotate to cold the corresponding ones and then move the cold ones to frozen, then all I will have to do, for backup, is:
- Upload recently warmed up ones
- Delete backup of buckets older than frozen time from long-term storage
If not, I will keep a directory for each warm and cold buckets and do the upload/rotation/deletion at every incremental backup round. On the restore, I will restore them by age from oldest to most recent, which is much more painful.
Thanks a lot
As per splunk documentation http://docs.splunk.com/Documentation/Splunk/6.4.0/Indexer/Backupindexeddata , if you will take backup of hot buckets then it will not useful when you will restore them. So you can take backup of Warm and Cold buckets.
To summarize: hot buckets - Currently being written to; do not back these up. warm buckets - Rolled from hot; can be safely backed up. cold buckets - Rolled from warm; buckets are moved to another location. frozen buckets - The indexer deletes these, but you can archive their contents first.
What are the consequences of running with too many buckets?
When I restore them in thawdb, is the data copied from thawdb to a new bucket or is it really staying in this directory?
If it stays and doesn't migrate, thawdb can become very big at a point.
Do you finally suggest to do the rotation manually when backuping so, when restoring, there is already the right amount of bucket?
If you are running Indexer Cluster and if you will have too many buckets then you might face issue which is mentioned in https://answers.splunk.com/answers/233441/cluster-master-is-unable-to-meet-search-factor-and.html
Hope this clarifies your query.
Thanks a lot!
Just one final question:
What the difference between restoring in thaweddb beween restoring in db/ please?
If you restore in thaweddb then you can use those buckets directly without restarting splunk but it will not remove when it will age.
If you restore in db folder then you need to reindex whole index but I have never tried to restore in db folder.