I'm working on doing some data cloning.
As a first step, outputs.conf (on a virgin 6.4.1 universal forwarder on Windows) looks like this, and all is well.
[tcpout] defaultGroup = default-autolb-group [tcpout:default-autolb-group] server = splunk-c-ix.local:9997
Data goes to splunk-c-ix just fine.
When I add another output group (even without making it the default or referring to it in any _TCP_ROUTING lines), then _internal output starts going to both groups.
[tcpout] defaultGroup = default-autolb-group [tcpout:default-autolb-group] server = splunk-c-ix.local:9997 [tcpout:clone-group] server = splunk-c-hf.local:9997
I dug into it, and found
_TCP_ROUTING = * inside the
[monitor://...] stanzas inside $SPLUNK_HOME\apps\SplunkUniversalForwarder\defaults\inputs.conf, which accounts for the behaviour.
I was hoping I could just do a blacklist for the _* indexes on the tcpout:clone-group, but the docs indicate that blacklist/whitelist only happens globally.
Is there an easy way to override this besides hunting down all the
_TCP_ROUTING = * in the inputs.conf and overriding them in a local\inputs.conf?
Have you tried to include this, in your system/local/inputs.conf
[default] _TCP_ROUTING = default-autolb-group
It should override all default settings, to send default inputs just to the default group.
This seems to work for most inputs but the _internal inputs remain unchanged this is output from
/opt/splunkforwarder/bin/splunk btool inputs list
_TCP_ROUTING = *
_rcvbuf = 1572864
host = myVeryPersonalForwarder
index = _internal
make sense that this wouldn't work. The _TCP_ROUTING in a [default] stanza would only be used if _TCP_ROUTING was not specified elsewhere, and _TCP_ROUTING is specified elsewhere, so the [default] one gets ignored.
I hadn't receive gfuente's suggestion yet. which looks promising, I will have to see if it can be adapted to a deployment server fed environment (system/local/inputs.conf is not something that can be distributed via DS). That may be the way to go.
Right now, I just added overrides to the _TCP_ROUTING for guilty [monitor://] stanzas in a deployed inputs.conf:
[monitor://C:\Program Files\SplunkUniversalForwarder\etc\splunk.version] _TCP_ROUTING = default-autolb-group [monitor://C:\Program Files\SplunkUniversalForwarder\var\log\splunk\metrics.log] _TCP_ROUTING = default-autolb-group [monitor://C:\Program Files\SplunkUniversalForwarder\var\log\splunk\splunkd.log] _TCP_ROUTING = default-autolb-group
It's ugly and a little brittle (will need to watch future versions to see if they add monitor: stanzas, and someone will break me sooner or later by deploying Splunk onto the 😧 drive), but it works.