Deployment Architecture

What do I do if rebuilding a bucket fails?

wrangler2x
Motivator

Splunk version 5.0.5, build 179365, Linux-i386

Following recovery from an unplanned power outage, I got the message "Error in 'databasePartitionPolicy': Failed to read 1 event(s) from rawdata in bucket 'exchange_index~497~E8A41E0F-9507-4F30-B283-B1E932EAA801'. Rawdata may be corrupt, see search.log" while doing a search in the GUI. I had previously run a 'splunk fsck --repair --all'.

Taking the time the search was running in, I got the epoch time and figured-out what bucket was involved. I then used 'splunk rebuild' to rebuild the bucket (with splunkd stopped). Here is the result:

$ splunk rebuild /newlog/splunkDB/exchange_index/db/db_1410451894_1410368671_497
terminate called after throwing an instance of 'JournalSliceDirectory::error'
  what():  Error reading compressed journal while streaming: gzip data truncated, provider=/newlog/splunkDB/exchange_index/db/db_1410451894_1410368671_497/rawdata/journal.gz
ERROR: pid 31071 terminated with signal 6 (core dumped)
Rebuilding bucket failed

I don't see anything in the documentation that indicates a next step if the bucket rebuild fails. I'd like to know if anyone has got a recommendation about next steps.

1 Solution

wrangler2x
Motivator

I managed to come up with one answer on my own. If anyone has a better answer, please post it because I have saved the bucket. Here is what I did...

  1. splunk stop splunkd
  2. splunk cmd exporttool /newlog/splunkDB/exchange_index/db/db_1410451894_1410368671_497 /tmp/db_1410451894_1410368671_497 -csv
  3. mv /newlog/splunkDB/exchange_index/db/db_1410451894_1410368671_497 /tmp/db_1410451894_1410368671_497-corrupt
  4. splunk cmd importtool /newlog/splunkDB/exchange_index/db/db_1410451894_1410368671_497 /tmp/db_1410451894_1410368671_497.csv
  5. ls /newlog/splunkDB/exchange_index/db/db_1410451894_1410368671_497
  6. verified that everything looked like it should.
  7. splunk start splunkd

The import gave this message:

Successfully imported 3438855 events into bucket.
Please ensure this bucket resides in a valid index and restart Splunk to recognize the new events.

The restart was normal, with no errors.

I was able to run the search that previously generated the "Error in 'databasePartitionPolicy'" error without any errors.

View solution in original post

wrangler2x
Motivator

I managed to come up with one answer on my own. If anyone has a better answer, please post it because I have saved the bucket. Here is what I did...

  1. splunk stop splunkd
  2. splunk cmd exporttool /newlog/splunkDB/exchange_index/db/db_1410451894_1410368671_497 /tmp/db_1410451894_1410368671_497 -csv
  3. mv /newlog/splunkDB/exchange_index/db/db_1410451894_1410368671_497 /tmp/db_1410451894_1410368671_497-corrupt
  4. splunk cmd importtool /newlog/splunkDB/exchange_index/db/db_1410451894_1410368671_497 /tmp/db_1410451894_1410368671_497.csv
  5. ls /newlog/splunkDB/exchange_index/db/db_1410451894_1410368671_497
  6. verified that everything looked like it should.
  7. splunk start splunkd

The import gave this message:

Successfully imported 3438855 events into bucket.
Please ensure this bucket resides in a valid index and restart Splunk to recognize the new events.

The restart was normal, with no errors.

I was able to run the search that previously generated the "Error in 'databasePartitionPolicy'" error without any errors.

cbowles
Explorer

This comment really saved me a lot of trouble, I can verify that these steps works.

Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...