Deployment Architecture

Thawing data in an indexer clustering environment, do I need the rb_ buckets too?

dfronck
Communicator

Our Splunk implementation uses a cluster, so we have db_nnnnnn and rb_nnnnnn files in the frozendb directory. The rb directories are the replicated buckets.

Do I have to restore the db and rb directories?

The doc says to do the following.

cp -r db_1181756465_1162600547_1001 $SPLUNK_HOME/var/lib/splunk/defaultdb/thaweddb
splunk rebuild $SPLUNK_HOME/var/lib/splunk/defaultdb/thaweddb/db_1181756465_1162600547_1001
splunk restart

The problem is, if I only restore the db files, I get the following errors in the Master Node on the Fixup Tasks - Pending screen.

Search Factor: cannot fix up search factor as bucket is not serviceable
Replication Factor: cannot replicate as bucket is not serviceable
Generation: cannot fix up search factor as bucket is not serviceable;

If I also restore the rb files, I do not get those errors, but I'd think that I might get duplicate data.

1 Solution

dxu_splunk
Splunk Employee
Splunk Employee

"bucket is not serviceable" is a temporary state - we don't immediately start replicating / fixing up buckets, instead we'll wait a few minutes before we start doing the fixups.

if you restore all copies of a buckets' db_ and rb_ versions - it won't show the "bucket is not serviceable" message because theres no replications that we need to do for it, so it'll skip the temporary not serviceable state.

you'll have duplicate data, but thats the point of clustering - it'll duplicate your data :). it'll only have one copy of the data return results from a search however (the primary copy)

edit: there is a bug with thawing buckets into the thawed folder:
http://answers.splunk.com/answers/153341/thawed-buckets-error-clusterslavebuckethandler-failed-to-tr...

the cluster has trouble replicating these buckets (but it will be made primary if its searchable). workaround is to thaw them back into the db/ or colddb/ folders, where we can then replicate them - or thaw all copies so replication is not needed.

View solution in original post

splunkreal
Motivator

Hello guys,

could you confirm we should copy thawed buckets into colddb instead of thaweddb in 6.5.2?

Thanks.

* If this helps, please upvote or accept solution 🙂 *
0 Karma

lguinn2
Legend

I think the answer and the dialog is great, but for many cases I would ask: do you really need to restore the buckets to the same place they came from?

Here is a use case: After diagnosing a current security problem, you need to go back in time and see if certain patterns occurred back then. This is a one-shot search; after you do these forensics, you are done with the old data.

Instead of dragging this data back into your production environment, why not restore the frozen buckets to a test server, run the searches there and then delete them? All you would need to do is:

  1. Create an index on the test server to hold the restored data; let's call the index "forensics".
  2. Put the restored buckets in the thawed directory of the forensics index.
  3. Restart Splunk.
  4. Search the forensics index as needed.
  5. When finished, disable/delete the forensics index from the test server.

This will have no impact on your production environment, and you don't need to care if production is clustered or not.

dfronck
Communicator

I down-voted this because it really didn't have much to do with my question.

While it might a good idea, it doesn't consider a lot of issues that would come up. Off the top of my head, here's a couple of stopper issues for me. I'd have to copy terabytes of data from 7 indexers to a spare, multi-terabyte server that I really don't have.

0 Karma

dxu_splunk
Splunk Employee
Splunk Employee

"bucket is not serviceable" is a temporary state - we don't immediately start replicating / fixing up buckets, instead we'll wait a few minutes before we start doing the fixups.

if you restore all copies of a buckets' db_ and rb_ versions - it won't show the "bucket is not serviceable" message because theres no replications that we need to do for it, so it'll skip the temporary not serviceable state.

you'll have duplicate data, but thats the point of clustering - it'll duplicate your data :). it'll only have one copy of the data return results from a search however (the primary copy)

edit: there is a bug with thawing buckets into the thawed folder:
http://answers.splunk.com/answers/153341/thawed-buckets-error-clusterslavebuckethandler-failed-to-tr...

the cluster has trouble replicating these buckets (but it will be made primary if its searchable). workaround is to thaw them back into the db/ or colddb/ folders, where we can then replicate them - or thaw all copies so replication is not needed.

dfronck
Communicator

If you only restore the db files, "bucket is not serviceable" is a permanent state. Well, 20 hours at least so I assume never. I assume that's because they are in the thaweddb directory and the code probably says not to replicate that.

I restored all the rb files and the errors went away and I did not get dups. I assume because the Master Node is smarter than me and said, "don't search those rb files!"

0 Karma

dxu_splunk
Splunk Employee
Splunk Employee

oops, theres a bug with thawed bucket state. see http://answers.splunk.com/answers/153341/thawed-buckets-error-clusterslavebuckethandler-failed-to-tr... . your error went away since u thawed all copies of a bucket, so replication was no longer necessary.

will edit my answer accordingly

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...