What do I do if rebuilding a bucket fails?

[wrangler2x]

Splunk version 5.0.5, build 179365, Linux-i386

Following recovery from an unplanned power outage, I got the message "Error in 'databasePartitionPolicy': Failed to read 1 event(s) from rawdata in bucket 'exchange_index~497~E8A41E0F-9507-4F30-B283-B1E932EAA801'. Rawdata may be corrupt, see search.log" while doing a search in the GUI. I had previously run a 'splunk fsck --repair --all'.

Taking the time the search was running in, I got the epoch time and figured-out what bucket was involved. I then used 'splunk rebuild' to rebuild the bucket (with splunkd stopped). Here is the result:

$ splunk rebuild /newlog/splunkDB/exchange_index/db/db_1410451894_1410368671_497
terminate called after throwing an instance of 'JournalSliceDirectory::error'
  what():  Error reading compressed journal while streaming: gzip data truncated, provider=/newlog/splunkDB/exchange_index/db/db_1410451894_1410368671_497/rawdata/journal.gz
ERROR: pid 31071 terminated with signal 6 (core dumped)
Rebuilding bucket failed

I don't see anything in the documentation that indicates a next step if the bucket rebuild fails. I'd like to know if anyone has got a recommendation about next steps.

[wrangler2x · Accepted solution]

I managed to come up with one answer on my own. If anyone has a better answer, please post it because I have saved the bucket. Here is what I did...

1. splunk stop splunkd
2. splunk cmd exporttool /newlog/splunkDB/exchange_index/db/db_1410451894_1410368671_497 /tmp/db_1410451894_1410368671_497 -csv
3. mv /newlog/splunkDB/exchange_index/db/db_1410451894_1410368671_497 /tmp/db_1410451894_1410368671_497-corrupt
4. splunk cmd importtool /newlog/splunkDB/exchange_index/db/db_1410451894_1410368671_497 /tmp/db_1410451894_1410368671_497.csv
5. ls /newlog/splunkDB/exchange_index/db/db_1410451894_1410368671_497
6. verified that everything looked like it should.
7. splunk start splunkd

The import gave this message:

Successfully imported 3438855 events into bucket.
Please ensure this bucket resides in a valid index and restart Splunk to recognize the new events.

The restart was normal, with no errors.

I was able to run the search that previously generated the "Error in 'databasePartitionPolicy'" error without any errors.

[cbowles]

This comment really saved me a lot of trouble, I can verify that these steps works.