Trying to fix the corrupted bucket. Error - JournalSliceDirectory: Cannot seek to rawdata offset 0

[amitm05]

Hi,
I am getting the error:

JournalSliceDirectory: Cannot seek to rawdata offset 0, path="/opt/splunk/var/li b/splunk/indextest/db/<bucket_id>/rawdata"

I understand this means that the bucket is corrupted. I confirmed this by running the Splunk fsck scan and got the same bucket flagged as corrupted. Now I am trying to rebuild this bucket by Splunk rebuild and Splunk fsck repair commands but still not able to.

I further tried to decompress/open my journal.gz of the corrupted directory and I am getting the error that its corrupted and cannot be opened. Now I've got this problem on a single indexer env and there are no other copies of the bucket available.

Can someone point out how this can be fixed?

[amitm05 · Accepted solution]

Sorted.
Downloaded the journal file from the server. Decompressed it using 7z. Then recompressed to gz. Put it back in the bucket. And restarted splunk.

[effem]

Isn't that exactly what is suggested in the link I posted?

[amitm05]

No. with gunzip it wasnt working. I wasnt even able to get ahead of the first step. The second command was throwing error only.
May be the point is that 7z can also help, but ofcourse you'll have to choose gz while recompressing it back because that is what splunk expects.

[DavidHourani]

Hi @amitm05,

Have you tried running fsck for repair ? You can follow this guide for repairing buckets in standalone indexers :
https://docs.splunk.com/Documentation/Splunk/latest/Indexer/Bucketissues

And here you can find more options and parameters for the fsck command:
https://docs.splunk.com/Documentation/Splunk/latest/Troubleshooting/CommandlinetoolsforusewithSuppor...

Let me know if that helps.

Cheers,
David

[amitm05]

Hi David
Tried these but nope. Additionally I tried the exporttool to csv and then import back to reconstruct the bucket. But its failing to read the journal at all

[effem]

May be that this helps.
Although it sounds like you tried these steps already?
https://answers.splunk.com/answers/389363/getting-error-streamed-search-execute-failed-becau.html

[amitm05]

yes, I've tried these steps already but its still not able to fix the jornal.gz