topic Is there a way to restore hot buckets? in Deployment Architecture

Is there a way to restore hot buckets?

Michael — Thu, 24 Mar 2011 04:31:21 GMT

Nope, didn't know about the proper procedure for backing up database and NOT being able to backup hot buckets (now I know: roll to warm, backup the warm...etc.).

Ok, assuming we thought we were getting backups, and we have data backed up from the hot buckets (or so it appears; about 17 gig worth). We have no warm or cold buckets backed up.

Are we completely hosed now that we had a catastrophic fail/rebuild of our main server?

Is there any way to recover the data from what appears to be backed up hot-buckets? We've tried various forms of voodoo and yoga poses, but none work so far...

Thanks, look forward to the news (DSS inspection starts Monday, updating my resume this evening...).

Re: Is there a way to restore hot buckets?

dwaddle — Thu, 24 Mar 2011 04:51:05 GMT

I don't have a real answer for this, but would recommend opening a support case. If anyone has a chance of patching your buckets to make them usable, the skill would be there.

Re: Is there a way to restore hot buckets?

Genti — Fri, 25 Mar 2011 04:30:31 GMT

Michael,
this might probably best handled by splunk support but here is a quick response:

If you have the data backed up, as you seem to be saying - you have 17 gigs of HOT buckets - then you should have some directories that look like this:

hot_v1_2
hot_v1_3
hot_v1_4
hot_v1_5
hot_v1_6

Then if you have these directories all you should need to do is install the EXACT same version of splunk you had prior to fail
Create a new index, call it BACKUP and STOP splunk. Then browse to /splunk/var/lib/splunk/BACKUP/db/ and paste the above hot directories. Make absolutely sure that no hot buckets have the same id (the id is the 2-6 number)
Start splunk and you SHOULD be able to see your old data..

Re: Is there a way to restore hot buckets?

Michael — Tue, 29 Mar 2011 02:12:17 GMT

Thanks Genti for the idea. Here's what we ended up doing. Using a separate box (not our main indexer) we installed the same version of Splunk.

Splunk stop.

Copied the hot_v1_1, etc. directories into ../splunk/var/lib/splunk/defaultdb/db.

Then I manually edited .metaManifest to make sure there was a line for each instance of hot directory, i.e.:

/opt/splunk/var/lib/splunk/defaultdb/db/hot_v1_1 /opt/splunk/var/lib/splunk/defaultdb/db/hot_v1_2 etc.

Then edited .bucketManifest to include a line for each hot directory (these apparently get transformed by Splunk once you start it back up):

i.e.: 1 : hot_v1_1 2 : hot_v1_2

Splunk start.

After their conversion, they looked something like:

1 : db_1296702713_1295776921_1 2 : db_1296397258_1296021601_2

Worked like a champ!