How to fix BTree unexpected offset 0 while on order 1?

[yuanliu]

I'm getting these failures after a prior disk corruption.

ERROR TailReader [1876879 tailreader0] - Ignoring path="/somepath/somefile" due to: BTree::Exception: unexpected resulting offset 0 while on order 1, starting from offset 2056 node offset: 53209432 order: 255 keys: {  } children: { 0 }

I thought I had ejected buckets affected by the corruption; in addition, recent ingestions all go into buckets created after the corruption.

What can I do to fix this?

[yuanliu]

Indeed this error is related to ingestion failure.  I just thought the BTree problem is inside the buckets.  To clean up, do I just delete everything in fishbucket?  Reingestion is not a problem, but I do not want to cause other behavior changes.

[PickleRick · Accepted solution]

If you had bucket problems, the error would come from another component 🙂

Normally with stale fishbucket entries and similar problems you'd simply remove a particular entry from the fishbucket but in this case it looks that the fishbucket database itself is damaged so you probably need to remove whole fishbucket - stop splunkd, remove var/lib/splunk/fishbucket, start splunkd

You might want to fiddle with the db first. If I remember correctly (but I wouldn't bet any money on it), it's a berkeleydb so it might be possible to repair it.

[yuanliu]

Removing databases in fishbucket restored order in my ingestions.  Thank you!

For the record, I preserved fishbucket/db/ which does not contain any BTree. (Had I known this, I could have done this while cleaning up disk corruption.  There should be no assumption that fishbucket escaped corruption.)

[PickleRick]

This is a tailreader message so it should be about ingestion, not indexed data.

You might need to clean fishbucket (of course it will reingest all monitor inputs).