I've had complaints from customers that data is taking too long to appear in the system. Today, one of the Windows event forwarders was 2 hours behind... I looked for "blocked" messages, and see lots of typing queue and parsingqueue blocked messages. How can I troubleshoot these? Are there any settings that can be configured to help? (I'm at Splunk 6.1.9).
Sometimes this happens when your CPU or Memory Usage is too high on the forwarders, or even the indexers.
Check your CPU usage and load averages, etc. If they are too high, you may need to upgrade some systems.
Or maybe even you need to optimize some large searches, etc.
A common cause of this is monitoring a super large folder like this:
You're effectively telling splunk to index everything under /path thats a .log file but maybe there are billions of files in this directory that are .txt files, etc. Splunk has to keep a running list of what it's already processed, etc. and things get messy... especially on non-reference hardware such as 1 CPU virtual machines, etc.