Deployment Architecture

What do I do if rebuilding a bucket fails?

wrangler2x
Motivator

Splunk version 5.0.5, build 179365, Linux-i386

Following recovery from an unplanned power outage, I got the message "Error in 'databasePartitionPolicy': Failed to read 1 event(s) from rawdata in bucket 'exchange_index~497~E8A41E0F-9507-4F30-B283-B1E932EAA801'. Rawdata may be corrupt, see search.log" while doing a search in the GUI. I had previously run a 'splunk fsck --repair --all'.

Taking the time the search was running in, I got the epoch time and figured-out what bucket was involved. I then used 'splunk rebuild' to rebuild the bucket (with splunkd stopped). Here is the result:

$ splunk rebuild /newlog/splunkDB/exchange_index/db/db_1410451894_1410368671_497
terminate called after throwing an instance of 'JournalSliceDirectory::error'
  what():  Error reading compressed journal while streaming: gzip data truncated, provider=/newlog/splunkDB/exchange_index/db/db_1410451894_1410368671_497/rawdata/journal.gz
ERROR: pid 31071 terminated with signal 6 (core dumped)
Rebuilding bucket failed

I don't see anything in the documentation that indicates a next step if the bucket rebuild fails. I'd like to know if anyone has got a recommendation about next steps.

1 Solution

wrangler2x
Motivator

I managed to come up with one answer on my own. If anyone has a better answer, please post it because I have saved the bucket. Here is what I did...

  1. splunk stop splunkd
  2. splunk cmd exporttool /newlog/splunkDB/exchange_index/db/db_1410451894_1410368671_497 /tmp/db_1410451894_1410368671_497 -csv
  3. mv /newlog/splunkDB/exchange_index/db/db_1410451894_1410368671_497 /tmp/db_1410451894_1410368671_497-corrupt
  4. splunk cmd importtool /newlog/splunkDB/exchange_index/db/db_1410451894_1410368671_497 /tmp/db_1410451894_1410368671_497.csv
  5. ls /newlog/splunkDB/exchange_index/db/db_1410451894_1410368671_497
  6. verified that everything looked like it should.
  7. splunk start splunkd

The import gave this message:

Successfully imported 3438855 events into bucket.
Please ensure this bucket resides in a valid index and restart Splunk to recognize the new events.

The restart was normal, with no errors.

I was able to run the search that previously generated the "Error in 'databasePartitionPolicy'" error without any errors.

View solution in original post

wrangler2x
Motivator

I managed to come up with one answer on my own. If anyone has a better answer, please post it because I have saved the bucket. Here is what I did...

  1. splunk stop splunkd
  2. splunk cmd exporttool /newlog/splunkDB/exchange_index/db/db_1410451894_1410368671_497 /tmp/db_1410451894_1410368671_497 -csv
  3. mv /newlog/splunkDB/exchange_index/db/db_1410451894_1410368671_497 /tmp/db_1410451894_1410368671_497-corrupt
  4. splunk cmd importtool /newlog/splunkDB/exchange_index/db/db_1410451894_1410368671_497 /tmp/db_1410451894_1410368671_497.csv
  5. ls /newlog/splunkDB/exchange_index/db/db_1410451894_1410368671_497
  6. verified that everything looked like it should.
  7. splunk start splunkd

The import gave this message:

Successfully imported 3438855 events into bucket.
Please ensure this bucket resides in a valid index and restart Splunk to recognize the new events.

The restart was normal, with no errors.

I was able to run the search that previously generated the "Error in 'databasePartitionPolicy'" error without any errors.

cbowles
Explorer

This comment really saved me a lot of trouble, I can verify that these steps works.

Get Updates on the Splunk Community!

Splunk Forwarders and Forced Time Based Load Balancing

Splunk customers use universal forwarders to collect and send data to Splunk. A universal forwarder can send ...

NEW! Log Views in Splunk Observability Dashboards Gives Context From a Single Page

Today, Splunk Observability releases log views, a new feature for users to add their logs data from Splunk Log ...

Last Chance to Submit Your Paper For BSides Splunk - Deadline is August 12th!

Hello everyone! Don't wait to submit - The deadline is August 12th! We have truly missed the community so ...