How to troubleshoot why Windows event forwarders are reporting typing and parsingqueue blocked messages, causing delayed forwarding and indexing?

[a212830]

Hi,

I've had complaints from customers that data is taking too long to appear in the system. Today, one of the Windows event forwarders was 2 hours behind... I looked for "blocked" messages, and see lots of typing queue and parsingqueue blocked messages. How can I troubleshoot these? Are there any settings that can be configured to help? (I'm at Splunk 6.1.9).

[jkat54]

Sometimes this happens when your CPU or Memory Usage is too high on the forwarders, or even the indexers.

Check your CPU usage and load averages, etc. If they are too high, you may need to upgrade some systems.

Or maybe even you need to optimize some large searches, etc.

A common cause of this is monitoring a super large folder like this:

[monitor://path/.../*.log]

You're effectively telling splunk to index everything under /path thats a .log file but maybe there are billions of files in this directory that are .txt files, etc. Splunk has to keep a running list of what it's already processed, etc. and things get messy... especially on non-reference hardware such as 1 CPU virtual machines, etc.

[a212830]

Is this indicating a problem on the forwarders, or the indexers? I don't really see any way to determine that...

My indexers are loaded with resources - 48 cpu's and 256gb of memory, so I'd be very surprised if that's the issue. This particular forwarder processes event logs, at about 7 - 10k per minute.