topic Re: Universal Forwarder queue size in Getting Data In

Universal Forwarder queue size

naydenk — Mon, 28 Sep 2020 12:32:05 GMT

Hello
I have a Universal Forwarder that acts as an intermediary forwarder between about 200 other UFs and the Indexer. Data from the client UFs gets delayed or not sent at all and digging through the logs, I noticed this:

On the client UF:

09-29-2012 05:42:07.620 -0400 WARN TcpOutputFd - Connect to 10.xx.xx.60:9997 failed. No connection could be made because the target machine actively refused it.
09-29-2012 05:42:07.620 -0400 ERROR TcpOutputFd - Connection to host=10.xx.xx.60:9997 failed
09-29-2012 05:42:07.620 -0400 WARN TcpOutputProc - Applying quarantine to idx=10.xx.xx.60:9997 numberOfFailures=52

On the intermediary UF:

splunkd.log:
09-29-2012 06:35:48.722 -0400 INFO TailingProcessor - Could not send data to output queue (parsingQueue), retrying...
09-29-2012 06:35:49.504 -0400 INFO TailingProcessor - ...continuing.

metrics.log:
09-29-2012 06:27:24.509 -0400 INFO Metrics - group=queue, name=fschangemanager_queue, max_size_kb=5120, current_size_kb=0, current_size=0, largest_size=0, smallest_size=0
09-29-2012 06:27:24.509 -0400 INFO Metrics - group=queue, name=indexqueue, max_size_kb=10240, current_size_kb=0, current_size=0, largest_size=0, smallest_size=0
09-29-2012 06:27:24.509 -0400 INFO Metrics - group=queue, name=nullqueue, max_size_kb=10240, current_size_kb=0, current_size=0, largest_size=0, smallest_size=0
09-29-2012 06:27:24.509 -0400 INFO Metrics - group=queue, name=parsingqueue, blocked=true, max_size_kb=512, current_size_kb=511, current_size=409, largest_size=445, smallest_size=163
09-29-2012 06:27:24.509 -0400 INFO Metrics - group=queue, name=splunktcpin, max_size_kb=500, current_size_kb=423, current_size=218, largest_size=442, smallest_size=152
09-29-2012 06:27:24.509 -0400 INFO Metrics - group=queue, name=tcpin_queue, max_size_kb=10240, current_size_kb=0, current_size=0, largest_size=0, smallest_size=0
09-29-2012 06:27:24.509 -0400 INFO Metrics - group=queue, name=winparsing, max_size_kb=10240, current_size_kb=0, current_size=0, largest_size=1, smallest_size=0

I already updated the etc\system\local\server.conf on the intermediary UF adding this:

[queue]

maxSize = 10MB

[queue=parsingqueue]

maxSize = 10MB

[queue=splunktcpin]

maxSize = 10MB

however, as you can see from the log, those two queues are the only ones that seem to not get affected, showing max_size_kb to be 500-512 KB, instead of the 10 MB I set. Any idea why? Or am I not even on the right track?

Re: Universal Forwarder queue size

alacercogitatus — Sat, 29 Sep 2012 18:00:26 GMT

Looking at your metrics log, i see that the parsing queue is being blocked. Check here: http://splunk-base.splunk.com/answers/45676/what-causes-queues-on-indexer-to-block for some ideas on a blocked parsing queue. This is a little old, but http://wiki.splunk.com/Community:TroubleshootingBlockedQueues. And finally, http://splunk-base.splunk.com/answers/38218/universal-forwarder-parsingqueue-kb-size (notice how queue=parsingQueue [BIG Q] ) Might be case sensitive.

Re: Universal Forwarder queue size

naydenk — Sat, 29 Sep 2012 20:13:06 GMT

Thank you alacercogitatus - parsingQueue seems to have done the trick for that queue (I guess in the metrics.log the case is not correct). The only other one that I see still getting blocked relatively frequently is the splunktcpin, which also does not seem to get affected by the global [queue] stanza. I tried splunkTcpIn - that does not seem to be it. Any suggestions on what the case for this one would be? I can't seem to find documentation on that (other than all lower case)

Re: Universal Forwarder queue size

alacercogitatus — Sun, 30 Sep 2012 01:15:15 GMT

I can't seem to find the correct case either. Could be splunkTcpin, or SplunkTcpIn, I'd try a few different cases and see if any of them work. I'll keep looking as well.

Re: Universal Forwarder queue size

MuS — Tue, 02 Oct 2012 11:20:07 GMT

Hi, according to the docs it can be set in inputs.conf in the [TCP] Stanza http://docs.splunk.com/Documentation/Splunk/4.3.4/admin/Inputsconf

queueSize = [KB|MB|GB]
* Maximum size of the in-memory input queue.
* Defaults to 500KB.

Re: Universal Forwarder queue size

MuS — Tue, 02 Oct 2012 11:22:16 GMT

and if this does not work, then the correct name for the queue is 'tcpin_queue'

cheers,
MuS

Re: Universal Forwarder queue size

naydenk — Tue, 02 Oct 2012 19:56:47 GMT

Thank you MuS! It turns out the queue referred to as "splunktcpin" in the metrics.log is actually the splunk listener on port 9997, defined by .\etc\system\local\inputs.conf with this stanza

[splunktcp://9997]
queueSize = 10MB #<-- this is my change

The other one I had issues with, parsingQueue (which alacercogitatus helped me with), is defined in a different conf file - .\etc\system\local\server.conf, with:

[queue=parsingQueue]
maxSize = 30MB #<-- this is my change

A little confusing... 😞

Thank you both!