topic Re: Thawing data in an indexer clustering environment, do I need the rb_ buckets too? in Deployment Architecture

Thawing data in an indexer clustering environment, do I need the rb_ buckets too?

dfronck — Mon, 28 Sep 2020 20:24:52 GMT

Our Splunk implementation uses a cluster, so we have db_nnnnnn and rb_nnnnnn files in the frozendb directory. The rb directories are the replicated buckets.

Do I have to restore the db and rb directories?

The doc says to do the following.

cp -r db_1181756465_1162600547_1001 $SPLUNK_HOME/var/lib/splunk/defaultdb/thaweddb
splunk rebuild $SPLUNK_HOME/var/lib/splunk/defaultdb/thaweddb/db_1181756465_1162600547_1001
splunk restart

The problem is, if I only restore the db files, I get the following errors in the Master Node on the Fixup Tasks - Pending screen.

Search Factor: cannot fix up search factor as bucket is not serviceable
Replication Factor: cannot replicate as bucket is not serviceable
Generation: cannot fix up search factor as bucket is not serviceable;

If I also restore the rb files, I do not get those errors, but I'd think that I might get duplicate data.

Re: Thawing data in an indexer clustering environment, do I need the rb_ buckets too?

dxu_splunk — Tue, 30 Jun 2015 18:20:22 GMT

"bucket is not serviceable" is a temporary state - we don't immediately start replicating / fixing up buckets, instead we'll wait a few minutes before we start doing the fixups.

if you restore all copies of a buckets' db_ and rb_ versions - it won't show the "bucket is not serviceable" message because theres no replications that we need to do for it, so it'll skip the temporary not serviceable state.

you'll have duplicate data, but thats the point of clustering - it'll duplicate your data :). it'll only have one copy of the data return results from a search however (the primary copy)

edit: there is a bug with thawing buckets into the thawed folder:
http://answers.splunk.com/answers/153341/thawed-buckets-error-clusterslavebuckethandler-failed-to-trigger-replication

the cluster has trouble replicating these buckets (but it will be made primary if its searchable). workaround is to thaw them back into the db/ or colddb/ folders, where we can then replicate them - or thaw all copies so replication is not needed.

Re: Thawing data in an indexer clustering environment, do I need the rb_ buckets too?

dfronck — Tue, 30 Jun 2015 18:35:34 GMT

If you only restore the db files, "bucket is not serviceable" is a permanent state. Well, 20 hours at least so I assume never. I assume that's because they are in the thaweddb directory and the code probably says not to replicate that.

I restored all the rb files and the errors went away and I did not get dups. I assume because the Master Node is smarter than me and said, "don't search those rb files!"

Re: Thawing data in an indexer clustering environment, do I need the rb_ buckets too?

lguinn2 — Tue, 18 Aug 2015 00:11:43 GMT

I think the answer and the dialog is great, but for many cases I would ask: do you really need to restore the buckets to the same place they came from?

Here is a use case: After diagnosing a current security problem, you need to go back in time and see if certain patterns occurred back then. This is a one-shot search; after you do these forensics, you are done with the old data.

Instead of dragging this data back into your production environment, why not restore the frozen buckets to a test server, run the searches there and then delete them? All you would need to do is:

Create an index on the test server to hold the restored data; let's call the index "forensics".
Put the restored buckets in the thawed directory of the forensics index.
Restart Splunk.
Search the forensics index as needed.
When finished, disable/delete the forensics index from the test server.

This will have no impact on your production environment, and you don't need to care if production is clustered or not.

Re: Thawing data in an indexer clustering environment, do I need the rb_ buckets too?

dxu_splunk — Tue, 18 Aug 2015 00:33:01 GMT

oops, theres a bug with thawed bucket state. see http://answers.splunk.com/answers/153341/thawed-buckets-error-clusterslavebuckethandler-failed-to-trigger-replication . your error went away since u thawed all copies of a bucket, so replication was no longer necessary.

will edit my answer accordingly

Re: Thawing data in an indexer clustering environment, do I need the rb_ buckets too?

dfronck — Tue, 06 Oct 2015 18:39:03 GMT

I down-voted this because it really didn't have much to do with my question.

While it might a good idea, it doesn't consider a lot of issues that would come up. Off the top of my head, here's a couple of stopper issues for me. I'd have to copy terabytes of data from 7 indexers to a spare, multi-terabyte server that I really don't have.

Re: Thawing data in an indexer clustering environment, do I need the rb_ buckets too?

splunkreal — Sat, 19 May 2018 11:15:02 GMT

Hello guys,

could you confirm we should copy thawed buckets into colddb instead of thaweddb in 6.5.2?

Thanks.