Splunk version 5.0.5, build 179365, Linux-i386
Following recovery from an unplanned power outage, I got the message "Error in 'databasePartitionPolicy': Failed to read 1 event(s) from rawdata in bucket 'exchange_index~497~E8A41E0F-9507-4F30-B283-B1E932EAA801'. Rawdata may be corrupt, see search.log" while doing a search in the GUI. I had previously run a 'splunk fsck --repair --all'.
Taking the time the search was running in, I got the epoch time and figured-out what bucket was involved. I then used 'splunk rebuild' to rebuild the bucket (with splunkd stopped). Here is the result:
$ splunk rebuild /newlog/splunkDB/exchange_index/db/db_1410451894_1410368671_497 terminate called after throwing an instance of 'JournalSliceDirectory::error' what(): Error reading compressed journal while streaming: gzip data truncated, provider=/newlog/splunkDB/exchange_index/db/db_1410451894_1410368671_497/rawdata/journal.gz ERROR: pid 31071 terminated with signal 6 (core dumped) Rebuilding bucket failed
I don't see anything in the documentation that indicates a next step if the bucket rebuild fails. I'd like to know if anyone has got a recommendation about next steps.
I managed to come up with one answer on my own. If anyone has a better answer, please post it because I have saved the bucket. Here is what I did...
The import gave this message:
Successfully imported 3438855 events into bucket.
Please ensure this bucket resides in a valid index and restart Splunk to recognize the new events.
The restart was normal, with no errors.
I was able to run the search that previously generated the "Error in 'databasePartitionPolicy'" error without any errors.