You are most likely running into an issue with how Splunk deals with its whitelist and blacklist. Below are the default settings which are causing your conflict, pulled from: etc/system/default/outputs.conf
forwardedindex.0.whitelist = .*
forwardedindex.1.blacklist = _.*
forwardedindex.2.whitelist = (_audit|_internal|_introspection)
forwardedindex.filter.disable = false
The rules above dictate: rule #1( forwardedindex.1.blacklist = .* ) does successfully block all indexes that begin with an ""; however, rule #2 ( forwardedindex.2.whitelist = (_audit|_internal|_introspection) ) then tells Splunk to overwrites rule #1 for those particular indexes.
The easiest way to solve your issue would be to set your custom outputs.conf to one of the two below:
Rewrite rule #2 to remove _audit:
forwardedindex.2.whitelist = (_internal|_introspection)
Add a new rule #3 re-enforcing the blacklist of _audit:
forwardedindex.3.blacklist = _audit
Either one of these should work. Cheers!
... View more
In a short summary, this is expected behavior. While your default group has two groups, Splunk treats this as one, if you will, pipe. So if one of the connections errors out, it will stall both connections.
If you really need multiple outputs like this, best practices is to install another UF on the box and tcpout there also. Then you wont have any queue related blocking issues if one tcpout is backed up.
... View more