topic How to fix BTree unexpected offset 0 while on order 1? in Getting Data In

How to fix BTree unexpected offset 0 while on order 1?

yuanliu — Sat, 29 Jun 2024 00:36:53 GMT

I'm getting these failures after a prior disk corruption.

ERROR TailReader [1876879 tailreader0] - Ignoring path="/somepath/somefile" due to: BTree::Exception: unexpected resulting offset 0 while on order 1, starting from offset 2056 node offset: 53209432 order: 255 keys: { } children: { 0 }

I thought I had ejected buckets affected by the corruption; in addition, recent ingestions all go into buckets created after the corruption.

What can I do to fix this?

Re: How to fix BTree unexpected offset 0 while on order 1?

PickleRick — Sat, 29 Jun 2024 19:50:39 GMT

This is a tailreader message so it should be about ingestion, not indexed data.

You might need to clean fishbucket (of course it will reingest all monitor inputs).

Re: How to fix BTree unexpected offset 0 while on order 1?

yuanliu — Sun, 30 Jun 2024 05:15:40 GMT

Indeed this error is related to ingestion failure. I just thought the BTree problem is inside the buckets. To clean up, do I just delete everything in fishbucket? Reingestion is not a problem, but I do not want to cause other behavior changes.

Re: How to fix BTree unexpected offset 0 while on order 1?

PickleRick — Sun, 30 Jun 2024 07:15:28 GMT

If you had bucket problems, the error would come from another component 🙂

Normally with stale fishbucket entries and similar problems you'd simply remove a particular entry from the fishbucket but in this case it looks that the fishbucket database itself is damaged so you probably need to remove whole fishbucket - stop splunkd, remove var/lib/splunk/fishbucket, start splunkd

You might want to fiddle with the db first. If I remember correctly (but I wouldn't bet any money on it), it's a berkeleydb so it might be possible to repair it.

Re: How to fix BTree unexpected offset 0 while on order 1?

yuanliu — Tue, 02 Jul 2024 23:26:34 GMT

Removing databases in fishbucket restored order in my ingestions. Thank you!

For the record, I preserved fishbucket/db/ which does not contain any BTree. (Had I known this, I could have done this while cleaning up disk corruption. There should be no assumption that fishbucket escaped corruption.)