Monitoring Splunk

Splunk universal forwarer thruput and internal Splunk log

lukasmecir
Path Finder

Hello,

I have question about [thruput] setting on UF and internal Splunk log:

I did some tests with Splunk UF - I needed to simulate a problem with the tcpout queue and therefore I reduced the value of the parameter
[thruput]
maxKBps = <integer>
in the limits.conf file to low KBps values (eg 3KBps). UF is set to send its internal logs to IDX. However, I noticed that with such a low value of this parameter, UF stopped sending its internal metric logs (ie the contents of the $ SPLUNK_HOME/var/log/splunk/metrics.log file) to IDX. Logs were further written to the $ SPLUNK_HOME/var/log/splunk/metrics.log file, but were not sent to IDX. Is this normal behavior? It looks as if there is a mechanism that prioritizes the data collected over internal Splunk logs and suppresses the sending of internal Splunk logs to IDX - is it really so, is there such a mechanism?
I tried to find something about it in the documentation, but without success. Thank you in advance for any information.

Best regards

Lukas Mecir

0 Karma

lukasmecir
Path Finder

Hi @venkatasri 

thank for your input. I set thruput setting in /system/local/limits.conf and observed behavior as described in my first post. To be clear - I do not complain about UF stopped sending its internal metric logs with low thruput. In fact, from my point of view it makes sense. I would just like someone who knows things to confirm that there is really such a mechanism in Spluk, and that this is therefore expected and correct behavior.

0 Karma

venkatasri
SplunkTrust
SplunkTrust

Hi @lukasmecir 

There is no explicit mention of which inputs conf stanzas takes priority when thruput is low. thruput on UF default is 256. I believe it should still be ingesting metrics logs but at very slow rate and as you know they are under /system/default. 

Your custom data inputs could be under /system/local or /app/local those are precedence over /default where splunk _internal logs being set to monitor.

You can try moving custom data inputs conf to /system/default and similarly move metrics related conf to system or app/local and give a try. Try this command ./splunk list inputstatus to find the reason/where they have left to monitor.

---

An upvote would be appreciated and Accept solution if it helps!

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...