I have an Windows 2008R2 AD Domain Controller with the Splunk Universal Forwarder (v4.3.1) installed. Over the past few weeks the log level on this system increased 2-3 times (another mystery for now) and the events were getting further and further delayed. Searching the logs, I could see blocking happening on this system in teh "wel_queueu"
06-07-2012 09:53:40.275 -0400 INFO Metrics - group=queue, name=wel_queue, blocked=true, max_size_kb=500, current_size_kb=499, current_size=363, largest_size=391, smallest_size=162
I can't find reference to the wel_queue anywhere, but this is obviously an issue. I created a local limits.conf file on the DC under the SplunkUniversalForwarder app to change the maxKBps value from the default of 256 to 8192 and validated the settings took.
splunk cmd btool limits list --debug
...
SplunkUniv [thruput]
SplunkUniv maxKBps = 8192
...
Since making the change I haven't seen any more blocking errors, my latency is still several hours behind and is only gaining about 8 minutes per hour. The network view on the Windows Resource Monitor shows that the forwarder is sending about 120k-140k B/sec which isn't even hitting the 256KBps default... is there anything else I can do to speed up processing on this guy?
