topic Re: Splunk universal forwarer thruput and internal Splunk log in Monitoring Splunk

Splunk universal forwarer thruput and internal Splunk log

lukasmecir — Tue, 29 Jun 2021 11:22:56 GMT

Hello,

I have question about [thruput] setting on UF and internal Splunk log:

I did some tests with Splunk UF - I needed to simulate a problem with the tcpout queue and therefore I reduced the value of the parameter
[thruput]
maxKBps = <integer>
in the limits.conf file to low KBps values (eg 3KBps). UF is set to send its internal logs to IDX. However, I noticed that with such a low value of this parameter, UF stopped sending its internal metric logs (ie the contents of the $ SPLUNK_HOME/var/log/splunk/metrics.log file) to IDX. Logs were further written to the $ SPLUNK_HOME/var/log/splunk/metrics.log file, but were not sent to IDX. Is this normal behavior? It looks as if there is a mechanism that prioritizes the data collected over internal Splunk logs and suppresses the sending of internal Splunk logs to IDX - is it really so, is there such a mechanism?
I tried to find something about it in the documentation, but without success. Thank you in advance for any information.

Best regards

Lukas Mecir

Re: Splunk universal forwarer thruput and internal Splunk log

venkatasri — Thu, 01 Jul 2021 01:37:56 GMT

Hi @lukasmecir

There is no explicit mention of which inputs conf stanzas takes priority when thruput is low. thruput on UF default is 256. I believe it should still be ingesting metrics logs but at very slow rate and as you know they are under /system/default.

Your custom data inputs could be under /system/local or /app/local those are precedence over /default where splunk _internal logs being set to monitor.

You can try moving custom data inputs conf to /system/default and similarly move metrics related conf to system or app/local and give a try. Try this command ./splunk list inputstatus to find the reason/where they have left to monitor.

---

An upvote would be appreciated and Accept solution if it helps!

Re: Splunk universal forwarer thruput and internal Splunk log

lukasmecir — Wed, 07 Jul 2021 09:07:42 GMT

Hi @venkatasri

thank for your input. I set thruput setting in /system/local/limits.conf and observed behavior as described in my first post. To be clear - I do not complain about UF stopped sending its internal metric logs with low thruput. In fact, from my point of view it makes sense. I would just like someone who knows things to confirm that there is really such a mechanism in Spluk, and that this is therefore expected and correct behavior.