Splunk Errors

[sriva6]

Hi, I am getting this error when I open one of my dashboards today.

" Error in 'databasePartitionPolicy': Failed to read 1 event(s) from rawdata in bucket 'main-xxxxxx'. Rawdata may be corrupt, see search.log."

this is what i see in search.log

02-26-2013 11:22:21.540 INFO DispatchCommand - Round Robin Threaded ProviderQueue: done reading from peer 'BP1LCSAP031'
02-26-2013 11:22:23.506 ERROR JournalSlice - Cannot seek to 74529344
02-26-2013 11:22:23.506 ERROR databasePartitionPolicy - Failed to read event at address=2329042 in rawdata directory: \reuxeuss019-f07\splunk_index\defaultdb\db\db_1361833650_1361568580_55\rawdata
02-26-2013 11:22:23.506 ERROR databasePartitionPolicy - Failed to read 1 event(s) from rawdata in bucket 'main~55~004CC9C7-AEAA-4C5A-B3C7-2B22F4A91F7D'. Rawdata may be corrupt, see search.log
02-26-2013 11:22:23.521 INFO IndexScopedSearch - PREAD_HISTOGRAM: usec_1_8=3718 usec_8_64=0 usec_64_512=0 usec_512_4096=0 usec_4096_32768=9

Any suggestions please?

[Drainy · Accepted solution]

You may need to manually run FSCK against your buckets, have a look here for the detail;
http://docs.splunk.com/Documentation/Splunk/latest/Indexer/HowSplunkstoresindexes#Troubleshoot_your_...

Also, if you store your buckets on another filesystem/partition make sure that there are no issues with permissions or the user that Splunk is running as can access them still.

[sriva6]

running FSCK helped

[sriva6]

No, I haven't tried a reboot yet but this was working fine till yesterday. Also, I see these as well in the indexing errors:

INFO databasePartitionPolicy - idx=_audit Moving from='hot_v1_48' to warm='write error on hot bucket'
» 2/26/13
11:46:04.961 AM
02-26-2013 11:46:04.961 +0000 ERROR databasePartitionPolicy - Unable to write raw: for idx=_audit, path='\reuxeuss019-f07\splunk_index\audit\db\hot_v1_48'
» 2/26/13
11:45:26.989 AM
02-26-2013 11:45:26.989 +0000 INFO databasePartitionPolicy - idx=_internal Moving from='hot_v1_67' to warm='write error on hot bucket'

[SplunkFu]

tried a reboot of splunkd? this may rebuild corrupt sections.