I have only started using splunk on a test server, and I am consistently getting "skipped indexing of internal audit event will keep dropping events until indexer congestion is remedied. Check disk space and other issues that may cause indexer to block." The volume with the databases has 14 GB free. licensing states that I have only used 5% of quota. Restarting splunk did not seem to help either.
Any assistance would be greatly appreciated.
yannk mentions the SOS app - this has been deprecated - if you have version 6.3 or above this has been replaced by the DMC. We too are getting this error message - any thoughts on a solution for this issue?
Audit event generator: Now skipping indexing of internal audit events, because the downstream queue is not accepting data. Will keep dropping events until data flow resumes. Review system health: ensure downstream indexing and/or forwarding are operating correctly.
I am having this issue as well. It's on a SH which has some very full parsing queues. The other SHs on my Search Head Pool do not have full queues at all. All my Indexers' queues are all clear.
Suggestions on where to look next?
The exact message is a banner displayed on the search-head.
"Skipped indexing of internal audit event will keep dropping events until indexer congestion is remedied. Check disk space and other issues that may cause indexer to block"
The meaning of this message is that the indexers are busy, and the queues full.
Therefore the internal splunk logs (like audit) are disabled in order to dedicate all the performance to the indexing. "Your data is more important to us than our own logs"
If this is happening once a while, this may be peak of volume in your data, if this is happening constantly, this is a performance issue :
To troubleshoot, install the SOS app and check the "indexing performance" on your indexers. http://splunk-base.splunk.com/apps/29008/sos-splunk-on-splunk
The root causes can be