topic Re: JournalSliceDirectory: Cannot seek to rawdata offset 0, path="..." on running search in Splunk Web on Splunk non clustered indexer in Security

JournalSliceDirectory: Cannot seek to rawdata offset 0, path="..." on running search in Splunk Web on Splunk non clustered indexer

mouryagalla — Tue, 29 Sep 2020 16:02:48 GMT

I am using Splunk 6.6.2

When I ran search in Splunk Web for index for more than 30 days timeline "index="indextest" , I get this error:

JournalSliceDirectory: Cannot seek to rawdata offset 0, path="/opt/splunk/var/lib/splunk/indextest/db/db_1502353482_1504459082_1/rawdata'

I have gone through some answers posted in Splunk and tried few fsck commands to repair
i ran the fsck scan command identified the corrupted buckets:

Eg:
splunk scan --all-buckets-all-indexes

output in unix:
Operating on: idx=indextest bucket='/opt/splunk/var/lib//splunk/indextest/db/db_1502353482_1504459082_1/rawdata'

JournalSliceDirectory: Cannot seek to rawdata offset 0, path="/opt/splunk/var/li b/splunk/indextest/db/db_1502353482_1504459082_1/rawdata"

Corruption: corrupt slicesv2.dat or slices.dat

Then tried to repair them:
splunk repair --all-buckets-all-indexes

Eg:
splunk fsck repair --one-bucket --index-name=indextest--bucket-name=db_1502353482_1504459082_1 --try-warm-then-cold
output in unix:
Operating on: idx=indextest bucket='/opt/splunk/var/lib/splunk/indextest/db/db_1502353482_1504459082_1/'
(entire bucket) Rebuild for bucket='/opt/splunk/var/lib/splunk/indextest/db/db_1502353482_1504459082_1' took 64.23 milliseconds
Repair entire bucket, index=indextest, tryWarmThenCold=1, bucket=/opt/splunk/var/lib/splunk/indextest/db/db_1502353482_1504459082_1, exists=1, localrc=7, failReason=No bloomfilter in finalDir='/opt/splunk/var/lib/splunk/indextest/db/db_1502353482_1504459082_1'

The issue is not resolved.. Then

I even tried disabling the index

/opt/splunk/bin/splunk disable index name_of_your_index

I started splunk up and enabled the index from the web gui and restarted splunk

Still the issue is not resolved.

Any help and hints appreciated

Re: JournalSliceDirectory: Cannot seek to rawdata offset 0, path="..." on running search in Splunk Web on Splunk non clustered indexer

ddrillic — Thu, 28 Sep 2017 20:17:36 GMT

Re: JournalSliceDirectory: Cannot seek to rawdata offset 0, path="..." on running search in Splunk Web on Splunk non clustered indexer

mouryagalla — Tue, 29 Sep 2020 16:03:15 GMT

Thanks so much for your time and attention!

I even tried rebuilding.. It failed due to failReason=No bloomfilter which is same happened with fsck repair command.
I have only one indexer server in the architecture. Please find the following details in the corrupted and non corrupted buckets I have in my index.

Files in corrupted bucket:
[splunk@hostname db_1505749039_1505749029_0]$ ls
1505749039-1505749029-9561667152978923474.tsidx bloomfilter2 bucket_info.csv corrupt.all.marker Hosts.data rawdata Sources.data SourceTypes.data

Files in Non corrupted bucket:
[splunk@hostname db_1505804824_1505803018_1]$ ls
1505804824-1505803018-5429584547022512555.tsidx bloomfilter bucket_info.csv Hosts.data optimize.result rawdata Sources.data SourceTypes.data

Can I get any info on how i can fix the corrupted bucket by replacing the buckets from working ones? Will deleting the corrupted ones help? I have same issues with internal indexes even like main, _audit..

Re: JournalSliceDirectory: Cannot seek to rawdata offset 0, path="..." on running search in Splunk Web on Splunk non clustered indexer

mouryagalla — Tue, 29 Sep 2020 16:03:20 GMT

I fixed it now.. I replaced the contents of corrupted bucket with the non corrupted bucket of same index and ran the following cmd for the corrupted bucket.

splunk fsck repair --one-bucket --index-name=indextest--bucket-name=db_1502353482_1504459082_1 --try-warm-then-cold

Splunk repaired the corrupted index and the error is gone now.

Re: JournalSliceDirectory: Cannot seek to rawdata offset 0, path="..." on running search in Splunk Web on Splunk non clustered indexer

richgalloway — Fri, 29 Sep 2017 18:06:41 GMT

If you problem is resolved, please accept the answer to help future readers.