Wow, very thorough. 🙂
The TailingProcessor message means that it was unable to insert data into the parsingQueue, which, as you might guess, is where event parsing happens. This only happens when the queue is full - to confirm this, run 'grep blocked=true var/log/splunk/metrics.log*' and you should see many results.
Data travels through Splunk's queues pretty much linearly - meaning that if a queue further down the line (indexQueue) is blocked, it will eventually fill and block the queues that are feeding it (parsingQueue). The metrics.log output will indicate which queues are blocked.
The timeparsing warning doesn't have anything to do with the blocked queue message. Here are some things to check:
Does metrics.log on the indexer (receiver of the forwarded data) also contain blocked=true messages?
If so, the problem is likely on the indexing side only - perhaps the indexing box is overloaded (slow disks, etc).
If not: does metrics.log on the forwarder indicate that only the parsingQueue is blocked, or indexQueue as well?
Seeing only the parsingQueue would indicate that something along the lines of too much time spent running complex regexes on the data.
A blocked indexQueue would mean the data isn't being sent to the indexer fast enough. What type of link is between the forwarders and indexer? How much bandwidth are you actually seeing being used by the forwarding port?
Following the above steps and poking around a bit should point out the bottleneck fairly quickly.
... View more