Getting Data In

Splunk 6.2.3 Universal Forwarder maxQueueSize: What is the algorithm used to determine the amount of memory to use?

Path Finder

The outputs.conf.spec shows a default value of "auto". The Splunk Universal Forwarder version is 6.2.3 on RHEL 6.6. What is the algorithm used to determine the amount of memory to use? I have OS personnel asking what is the possible memory maximum for the agent.

0 Karma
1 Solution

Champion

The docs say

* If set to auto, chooses a value depending on whether useACK is enabled.
* If useACK=false, uses 500KB
* If useACK=true, uses 7MB

There is no algorithm, it uses one of two presets based on expected needs (an environment with acknowledgement enabled is probably going to need a bigger queue).

View solution in original post

Champion

The docs say

* If set to auto, chooses a value depending on whether useACK is enabled.
* If useACK=false, uses 500KB
* If useACK=true, uses 7MB

There is no algorithm, it uses one of two presets based on expected needs (an environment with acknowledgement enabled is probably going to need a bigger queue).

View solution in original post

Path Finder

This is good information. Do you know of 6.2.x or 6.3.x supports persistent output queues to disk vs. using more memory?

0 Karma

Champion

I don't think so. The only available persistent queues are used at the input stage (to not lose any input from a busy tcp input, for example) as described here. I don't think writing stuff to disk prior to indexing it is ever possible in splunk (except for the mentioned input queue) because of the principle it employs: your data will only ever "be" in one stage at a time really, so once you have "read" an input, the data is on its way to the indexer, passing any and all queues and pipelines. That means that if one of the stages is blocked or unavailable, your data will simply wait behind that stage (ultimately leading to a waiting file read or the need for a persistend input queue if it's a streaming input). That means intermediate persistency is unneccesary and could cause more problems than it solves.

0 Karma
State of Splunk Careers

Access the Splunk Careers Report to see real data that shows how Splunk mastery increases your value and job satisfaction.

Find out what your skills are worth!