The exact message is a banner displayed on the search-head.
"Skipped indexing of internal audit event will keep dropping events until indexer congestion is remedied. Check disk space and other issues that may cause indexer to block"
The meaning of this message is that the indexers are busy, and the queues full.
Therefore the internal splunk logs (like audit) are disabled in order to dedicate all the performance to the indexing. "Your data is more important to us than our own logs"
If this is happening once a while, this may be peak of volume in your data, if this is happening constantly, this is a performance issue :
To troubleshoot, install the SOS app and check the "indexing performance" on your indexers. http://splunk-base.splunk.com/apps/29008/sos-splunk-on-splunk
The root causes can be
- the forwarders are sending a large volume of events in burst -> you may change your monitoring or logrotation, and verify the thruput limit to spread the load over a longer period. (thruput in limits.conf)
- too large metadata files -> fixed in 5.*, upgrade the indexers.
- the indexer capacity is too low for your volume -> load balance over more indexers, faster cpu, faster disks
- the format of the data requires more index time processing (multiline events, invalid timestamp, linebreaking rules, wrong sourcetype, non optimized sourcetype) -> check your props and transforms, and the internal splunkd.log
- costly index time rules are setup (sedcmd, index time field extraction, complex routing rules, bad regexes)
-> check your props and transforms.
