Security

JournalSliceDirectory: Cannot seek to rawdata offset 0, path="..." on running search in Splunk Web on Splunk non clustered indexer

mouryagalla
Explorer

I am using Splunk 6.6.2

When I ran search in Splunk Web for index for more than 30 days timeline "index="indextest" , I get this error:

alt text

JournalSliceDirectory: Cannot seek to rawdata offset 0, path="/opt/splunk/var/lib/splunk/indextest/db/db_1502353482_1504459082_1/rawdata'

I have gone through some answers posted in Splunk and tried few fsck commands to repair
i ran the fsck scan command identified the corrupted buckets:

Eg:
splunk scan --all-buckets-all-indexes

output in unix:
Operating on: idx=indextest bucket='/opt/splunk/var/lib//splunk/indextest/db/db_1502353482_1504459082_1/rawdata'

JournalSliceDirectory: Cannot seek to rawdata offset 0, path="/opt/splunk/var/li b/splunk/indextest/db/db_1502353482_1504459082_1/rawdata"

Corruption: corrupt slicesv2.dat or slices.dat

Then tried to repair them:
splunk repair --all-buckets-all-indexes

Eg:
splunk fsck repair --one-bucket --index-name=indextest--bucket-name=db_1502353482_1504459082_1 --try-warm-then-cold
output in unix:
Operating on: idx=indextest bucket='/opt/splunk/var/lib/splunk/indextest/db/db_1502353482_1504459082_1/'
(entire bucket) Rebuild for bucket='/opt/splunk/var/lib/splunk/indextest/db/db_1502353482_1504459082_1' took 64.23 milliseconds
Repair entire bucket, index=indextest, tryWarmThenCold=1, bucket=/opt/splunk/var/lib/splunk/indextest/db/db_1502353482_1504459082_1, exists=1, localrc=7, failReason=No bloomfilter in finalDir='/opt/splunk/var/lib/splunk/indextest/db/db_1502353482_1504459082_1'

The issue is not resolved.. Then

I even tried disabling the index

/opt/splunk/bin/splunk disable index name_of_your_index

I started splunk up and enabled the index from the web gui and restarted splunk

Still the issue is not resolved.

Any help and hints appreciated

1 Solution

mouryagalla
Explorer

I fixed it now.. I replaced the contents of corrupted bucket with the non corrupted bucket of same index and ran the following cmd for the corrupted bucket.

splunk fsck repair --one-bucket --index-name=indextest--bucket-name=db_1502353482_1504459082_1 --try-warm-then-cold

Splunk repaired the corrupted index and the error is gone now.

View solution in original post

mouryagalla
Explorer

I fixed it now.. I replaced the contents of corrupted bucket with the non corrupted bucket of same index and ran the following cmd for the corrupted bucket.

splunk fsck repair --one-bucket --index-name=indextest--bucket-name=db_1502353482_1504459082_1 --try-warm-then-cold

Splunk repaired the corrupted index and the error is gone now.

richgalloway
SplunkTrust
SplunkTrust

If you problem is resolved, please accept the answer to help future readers.

---
If this reply helps you, Karma would be appreciated.
0 Karma

mouryagalla
Explorer

Thanks so much for your time and attention!

I even tried rebuilding.. It failed due to failReason=No bloomfilter which is same happened with fsck repair command.
I have only one indexer server in the architecture. Please find the following details in the corrupted and non corrupted buckets I have in my index.

Files in corrupted bucket:
[splunk@hostname db_1505749039_1505749029_0]$ ls
1505749039-1505749029-9561667152978923474.tsidx bloomfilter2 bucket_info.csv corrupt.all.marker Hosts.data rawdata Sources.data SourceTypes.data

Files in Non corrupted bucket:
[splunk@hostname db_1505804824_1505803018_1]$ ls
1505804824-1505803018-5429584547022512555.tsidx bloomfilter bucket_info.csv Hosts.data optimize.result rawdata Sources.data SourceTypes.data

Can I get any info on how i can fix the corrupted bucket by replacing the buckets from working ones? Will deleting the corrupted ones help? I have same issues with internal indexes even like main, _audit..

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...