topic Re: Why Splunk Web UI shows "Check space and other issues that may cause indexer to block" and "Rawdata may be corrupted"? in Getting Data In

Why Splunk Web UI shows "Check space and other issues that may cause indexer to block" and "Rawdata may be corrupted"?

Bizfinx_sysmon — Mon, 08 Sep 2014 10:10:53 GMT

Dear Support,

I have 2 messages on the Splunk web interface:
"skipped indexing of internal audit events will keep dropping events until indexer congestion is remedied. Check space and other issues that may caused indexer to block"
"Error in 'databasePartitionPolicy': Failure to read 1 event(s) from rawdata in bucket'_internal-1679-C10009BFA-1DE1-1A491-8895-E35E6F221168'. Rawdata maybe corrupted, see search.logs

I have tried to search under the following directory:
/opt/splunk/var/run/splunk/
where there are lots of directory with search.log.

I have tried to look at some of them and it seems to be ok.

Can someone from support advise on the above?

Re: Why Splunk Web UI shows "Check space and other issues that may cause indexer to block" and "Rawdata may be corrupted"?

grijhwani — Mon, 08 Sep 2014 13:53:49 GMT

Without looking at your specific setup, and/or having sight of specific error messages in the log, what you have told us above is suggestive of one of three things:

Your index disk is full (although experience teaches me to expect a different message)
Your index disk is experiencing errors which could be slowing throughput because of read/write retries (although again, if it is persistent I would expect a more explanatory error)
Your indexer is quite simply over-taxed (through memory exhaustion and excessive page swapping, or CPU exhaustion) or has reached the bandwidth cap you have set (or that it has defaulted to)

I'd be inclined to look for a system problem to begin with.

Re: Why Splunk Web UI shows "Check space and other issues that may cause indexer to block" and "Rawdata may be corrupted"?

Bizfinx_sysmon — Tue, 09 Sep 2014 06:40:36 GMT

Hi grijhwani,

Thanks for the update. FYI, I do not have such message anymore.

But I have a bigger issue now. I'm now not able to view my past data..
But I have check that the data seems to be there but Splunk is not able to link it back.

Can someone help?