Nope, didn't know about the proper procedure for backing up database and NOT being able to backup hot buckets (now I know: roll to warm, backup the warm...etc.).
Ok, assuming we thought we were getting backups, and we have data backed up from the hot buckets (or so it appears; about 17 gig worth). We have no warm or cold buckets backed up.
Are we completely hosed now that we had a catastrophic fail/rebuild of our main server?
Is there any way to recover the data from what appears to be backed up hot-buckets? We've tried various forms of voodoo and yoga poses, but none work so far...
Thanks, look forward to the news (DSS inspection starts Monday, updating my resume this evening...).
Thanks Genti for the idea. Here's what we ended up doing. Using a separate box (not our main indexer) we installed the same version of Splunk.
Splunk stop.
Copied the hot_v1_1, etc. directories into ../splunk/var/lib/splunk/defaultdb/db.
Then I manually edited .metaManifest to make sure there was a line for each instance of hot directory, i.e.:
/opt/splunk/var/lib/splunk/defaultdb/db/hot_v1_1 /opt/splunk/var/lib/splunk/defaultdb/db/hot_v1_2 etc.
Then edited .bucketManifest to include a line for each hot directory (these apparently get transformed by Splunk once you start it back up):
i.e.: 1 : hot_v1_1 2 : hot_v1_2
Splunk start.
After their conversion, they looked something like:
1 : db_1296702713_1295776921_1 2 : db_1296397258_1296021601_2
Worked like a champ!
Michael,
this might probably best handled by splunk support but here is a quick response:
If you have the data backed up, as you seem to be saying - you have 17 gigs of HOT buckets - then you should have some directories that look like this:
hot_v1_2
hot_v1_3
hot_v1_4
hot_v1_5
hot_v1_6
Then if you have these directories all you should need to do is install the EXACT same version of splunk you had prior to fail
Create a new index, call it BACKUP and STOP splunk.
Then browse to /splunk/var/lib/splunk/BACKUP/db/
and paste the above hot directories.
Make absolutely sure that no hot buckets have the same id (the id is the 2-6 number)
Start splunk and you SHOULD be able to see your old data..
I don't have a real answer for this, but would recommend opening a support case. If anyone has a chance of patching your buckets to make them usable, the skill would be there.