Nope, didn't know about the proper procedure for backing up database and NOT being able to backup hot buckets (now I know: roll to warm, backup the warm...etc.).
Ok, assuming we thought we were getting backups, and we have data backed up from the hot buckets (or so it appears; about 17 gig worth). We have no warm or cold buckets backed up.
Are we completely hosed now that we had a catastrophic fail/rebuild of our main server?
Is there any way to recover the data from what appears to be backed up hot-buckets? We've tried various forms of voodoo and yoga poses, but none work so far...
Thanks, look forward to the news (DSS inspection starts Monday, updating my resume this evening...).
I don't have a real answer for this, but would recommend opening a support case. If anyone has a chance of patching your buckets to make them usable, the skill would be there.
this might probably best handled by splunk support but here is a quick response:
If you have the data backed up, as you seem to be saying - you have 17 gigs of HOT buckets - then you should have some directories that look like this:
hot_v1_2 hot_v1_3 hot_v1_4 hot_v1_5 hot_v1_6
Then if you have these directories all you should need to do is install the EXACT same version of splunk you had prior to fail
Create a new index, call it BACKUP and STOP splunk. Then browse to
/splunk/var/lib/splunk/BACKUP/db/ and paste the above hot directories.
Make absolutely sure that no hot buckets have the same id (the id is the 2-6 number)
Start splunk and you SHOULD be able to see your old data..
Thanks Genti for the idea. Here's what we ended up doing. Using a separate box (not our main indexer) we installed the same version of Splunk.
Copied the hot_v1_1, etc. directories into ../splunk/var/lib/splunk/defaultdb/db.
Then I manually edited .metaManifest to make sure there was a line for each instance of hot directory, i.e.:
/opt/splunk/var/lib/splunk/defaultdb/db/hot_v1_1 /opt/splunk/var/lib/splunk/defaultdb/db/hot_v1_2 etc.
Then edited .bucketManifest to include a line for each hot directory (these apparently get transformed by Splunk once you start it back up):
i.e.: 1 : hot_v1_1 2 : hot_v1_2
After their conversion, they looked something like:
1 : db_1296702713_1295776921_1 2 : db_1296397258_1296021601_2
Worked like a champ!