Getting Data In

Trying to fix the corrupted bucket. Error - JournalSliceDirectory: Cannot seek to rawdata offset 0

amitm05
Builder

Hi,
I am getting the error:

JournalSliceDirectory: Cannot seek to rawdata offset 0, path="/opt/splunk/var/li b/splunk/indextest/db/<bucket_id>/rawdata"

I understand this means that the bucket is corrupted. I confirmed this by running the Splunk fsck scan and got the same bucket flagged as corrupted. Now I am trying to rebuild this bucket by Splunk rebuild and Splunk fsck repair commands but still not able to.

I further tried to decompress/open my journal.gz of the corrupted directory and I am getting the error that its corrupted and cannot be opened. Now I've got this problem on a single indexer env and there are no other copies of the bucket available.

Can someone point out how this can be fixed?

0 Karma
1 Solution

amitm05
Builder

Sorted.
Downloaded the journal file from the server. Decompressed it using 7z. Then recompressed to gz. Put it back in the bucket. And restarted splunk.

View solution in original post

amitm05
Builder

Sorted.
Downloaded the journal file from the server. Decompressed it using 7z. Then recompressed to gz. Put it back in the bucket. And restarted splunk.

effem
Communicator

Isn't that exactly what is suggested in the link I posted?

0 Karma

amitm05
Builder

No. with gunzip it wasnt working. I wasnt even able to get ahead of the first step. The second command was throwing error only.
May be the point is that 7z can also help, but ofcourse you'll have to choose gz while recompressing it back because that is what splunk expects.

0 Karma

DavidHourani
Super Champion

Hi @amitm05,

Have you tried running fsck for repair ? You can follow this guide for repairing buckets in standalone indexers :
https://docs.splunk.com/Documentation/Splunk/latest/Indexer/Bucketissues

And here you can find more options and parameters for the fsck command:
https://docs.splunk.com/Documentation/Splunk/latest/Troubleshooting/CommandlinetoolsforusewithSuppor...

Let me know if that helps.

Cheers,
David

0 Karma

amitm05
Builder

Hi David
Tried these but nope. Additionally I tried the exporttool to csv and then import back to reconstruct the bucket. But its failing to read the journal at all

0 Karma

effem
Communicator

May be that this helps.
Although it sounds like you tried these steps already?
https://answers.splunk.com/answers/389363/getting-error-streamed-search-execute-failed-becau.html

amitm05
Builder

yes, I've tried these steps already but its still not able to fix the jornal.gz

0 Karma
Get Updates on the Splunk Community!

Announcing Scheduled Export GA for Dashboard Studio

We're excited to announce the general availability of Scheduled Export for Dashboard Studio. Starting in ...

Extending Observability Content to Splunk Cloud

Watch Now!   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to leverage ...

More Control Over Your Monitoring Costs with Archived Metrics GA in US-AWS!

What if there was a way you could keep all the metrics data you need while saving on storage costs?This is now ...