I have some scripted inputs running on a few servers that will occasionally have very high system loads. The problem is I have holes in my scripted intervals during this time, when I need them the most. The forwarder doesn't die, it just seems to block sending due to limited system resources. I'd like for it NOT to do that, and fight for cycles so I can get a better glimpse into what is happening at this time from my scripted input. Any ideas on how to accomplish this?
This document is regarding streamfwd, but it details the default configuration of the universal forwarder's default output thruput limit, and how it can be tuned:
By default, the Splunk universal forwarder sends a maximum of 256 Kbps of data to indexers. Depending on your streamfwd configuration, your deployment might generate more data than this.
To modify or remove the default universal forwarder limit:
1. Edit $SPLUNK_HOME/etc/apps/SplunkUniversalForwarder/local/limits.conf.
2. Modify the [thruput] stanza. For example:
[thruput]
maxKBps = 0
could you expand on what "limited system resources" you're referring to?