I am using Splunk 6.6.2
When I ran search in Splunk Web for index for more than 30 days timeline "index="indextest" , I get this error:
JournalSliceDirectory: Cannot seek to rawdata offset 0, path="/opt/splunk/var/lib/splunk/indextest/db/db_1502353482_1504459082_1/rawdata'
I have gone through some answers posted in Splunk and tried few fsck commands to repair
i ran the fsck scan command identified the corrupted buckets:
Eg:
splunk scan --all-buckets-all-indexes
output in unix:
Operating on: idx=indextest bucket='/opt/splunk/var/lib//splunk/indextest/db/db_1502353482_1504459082_1/rawdata'
JournalSliceDirectory: Cannot seek to rawdata offset 0, path="/opt/splunk/var/li b/splunk/indextest/db/db_1502353482_1504459082_1/rawdata"
Corruption: corrupt slicesv2.dat or slices.dat
Then tried to repair them:
splunk repair --all-buckets-all-indexes
Eg:
splunk fsck repair --one-bucket --index-name=indextest--bucket-name=db_1502353482_1504459082_1 --try-warm-then-cold
output in unix:
Operating on: idx=indextest bucket='/opt/splunk/var/lib/splunk/indextest/db/db_1502353482_1504459082_1/'
(entire bucket) Rebuild for bucket='/opt/splunk/var/lib/splunk/indextest/db/db_1502353482_1504459082_1' took 64.23 milliseconds
Repair entire bucket, index=indextest, tryWarmThenCold=1, bucket=/opt/splunk/var/lib/splunk/indextest/db/db_1502353482_1504459082_1, exists=1, localrc=7, failReason=No bloomfilter in finalDir='/opt/splunk/var/lib/splunk/indextest/db/db_1502353482_1504459082_1'
The issue is not resolved.. Then
I even tried disabling the index
/opt/splunk/bin/splunk disable index name_of_your_index
I started splunk up and enabled the index from the web gui and restarted splunk
Still the issue is not resolved.
Any help and hints appreciated
I fixed it now.. I replaced the contents of corrupted bucket with the non corrupted bucket of same index and ran the following cmd for the corrupted bucket.
splunk fsck repair --one-bucket --index-name=indextest--bucket-name=db_1502353482_1504459082_1 --try-warm-then-cold
Splunk repaired the corrupted index and the error is gone now.
I fixed it now.. I replaced the contents of corrupted bucket with the non corrupted bucket of same index and ran the following cmd for the corrupted bucket.
splunk fsck repair --one-bucket --index-name=indextest--bucket-name=db_1502353482_1504459082_1 --try-warm-then-cold
Splunk repaired the corrupted index and the error is gone now.
If you problem is resolved, please accept the answer to help future readers.
Thanks so much for your time and attention!
I even tried rebuilding.. It failed due to failReason=No bloomfilter which is same happened with fsck repair command.
I have only one indexer server in the architecture. Please find the following details in the corrupted and non corrupted buckets I have in my index.
Files in corrupted bucket:
[splunk@hostname db_1505749039_1505749029_0]$ ls
1505749039-1505749029-9561667152978923474.tsidx bloomfilter2 bucket_info.csv corrupt.all.marker Hosts.data rawdata Sources.data SourceTypes.data
Files in Non corrupted bucket:
[splunk@hostname db_1505804824_1505803018_1]$ ls
1505804824-1505803018-5429584547022512555.tsidx bloomfilter bucket_info.csv Hosts.data optimize.result rawdata Sources.data SourceTypes.data
Can I get any info on how i can fix the corrupted bucket by replacing the buckets from working ones? Will deleting the corrupted ones help? I have same issues with internal indexes even like main, _audit..