_internal log tcp routing

[TheBravoSierra]

Splunk Universal Forwarder - Route wineventlog to one output group and _internal to another

I'm trying to configure a Universal Forwarder so that:

• wineventlog data is sent only to [tcpout:group1]
• _internal data is sent only to [tcpout:group2]

Currently, wineventlog is routing correctly to group1, but _internal events are being sent to both group1 and group2.

My goal is to stop _internal events from being forwarded to group1 and have them forwarded exclusively to group2.

I've already tried overriding _TCP_ROUTING=* by setting _TCP_ROUTING = group2 on both the monitor://...splunkd.log stanza and the [splunkd] stanza, but _internal data is still being forwarded to both output groups.

I've also reviewed Splunk's routing and filtering documentation, but I'm not sure what additional configuration is needed.

What is the correct way to exclude _internal events from group1 while continuing to send them to group2? Is there a props/transforms-based routing configuration or another setting I'm overlooking?

[Happy_16]

This isn't your config — it's a default the UF ships with. The internal-log monitor stanzas in $SPLUNK_HOME/etc/apps/SplunkUniversalForwarder/default/inputs.conf come with _TCP_ROUTING = *, which means "send to all tcpout groups." So the moment you added group2, _internal started cloning to both.

Your override was close but missed two things:

• [splunkd] is a sourcetype, not an input stanza — _TCP_ROUTING only works on the actual [monitor://...] stanza, so that line does nothing.
• _internal is fed by multiple monitors (splunkd.log, metrics.log, etc.), each with its own _TCP_ROUTING = *. You only overrode splunkd.log; the rest keep cloning to both groups.

(Side note: props/transforms routing won't work on a UF — it doesn't parse. _TCP_ROUTING in inputs.conf is the only lever. And internal indexes require an explicit _TCP_ROUTING, so override it to group2 — don't delete it.)

Fix. Enumerate what your version actually ships:

$SPLUNK_HOME/bin/splunk btool inputs list --debug | grep -i "TCP_ROUTING"

Then in $SPLUNK_HOME/etc/system/local/inputs.conf, override each internal stanza (header + routing line is enough; index/sourcetype still layer in from default):

[monitor://$SPLUNK_HOME/var/log/splunk/splunkd.log]
_TCP_ROUTING = group2

[monitor://$SPLUNK_HOME/var/log/splunk/metrics.log]
_TCP_ROUTING = group2

Add a block for every _internal stanza btool listed — any you skip keeps going to group1. Same applies to _audit (audit.log) and _introspection if you want those off group1 too. Restart, then re-run the btool grep to confirm they all resolve to group2.

---

If you meant something other than a reply to post, let me know.

[PickleRick]

OK. This needs a bit more explaining. Both you and @livehybrid are on the right track but there are some minute details which influence the overall ingestion mechanics.

1. Indeed, [splunkd] is the sourcetype stanza in props.conf. Yes, props.conf _can_ be used to override some metadata on distinct subsets of files ingested from a single monitored directory (it's called a "sourcetyper configuration") but it can only override the sourcetype and can only be used for the [source:..] stanza. Splunk uses this method by default to split all files read from the main splunkd log directory into separate sourcetypes (splunkd, splunkd_access and so on).

2. Another angle to this story is that Splunk cannot monitor a single file (meaning a single path leading to a file - if the file is symlinked, or under a bind mount, it can be accessed multiple times) using separate monitors. A file will be monitored using the "outmost" monitor stanza in such case. But. Some settings might get overwritten by parameters defined at the "lower level" monitors. So while the general $SPLUNK_HOME/var/log/splunk sends the data to index=_internal, specific subdirectories or files are destined to _configtracker, _introspection and so on.

From what I'm seeing in the configs (10.4), the only default input stanza which comes with explicit _TCP_ROUTING entries is monitor:///opt/splunk/etc/splunk.version (_TCP_ROUTING = *). All other inputs should inherit the default _TCP_ROUTING behaviour which is "send to the default group(s) as defined in outputs.conf").

One caveat - remember that wineventlog doesn't inherit (all) settings from the default stanza in inputs.conf.

So yes, your splunkd.log monitor stanza _might_ indeed be working (check for yourself - see if this particular file is being sent to all outputs, or are only the other files sent to that other output) but there are many more files in there and you'd need to either override the "global" monitor://opt/splunk/var/log/splunk to send to your preferred route only (but be aware that there are more directories in /opt/splunk/var/log and some of them are monitored with respective inputs.conf stanza monitors.

Having said that - if you are only receiving wineventlog data (which you want to be sent to one output group) and internal logs (which you want in the other one), the easiest way, as I said before, would be to set the default output group to the "internal" one and only override _TCP_ROUTING on the wineventlog inputs (or even on the default [WinEventLog://] stanza -  I can never remember how many slashes are there at the end of the stanza definition)

[livehybrid]

Hi @TheBravoSierra 

Im not convinced that adding a monitor://...splunkd.log stanza would overwrite the default "monitor://$SPLUNK_HOME/var/log/splunk" which picks up these files usually.

Could you try with the following?

[monitor://$SPLUNK_HOME/var/log/splunk]
_TCP_ROUTING = group2

🌟 Did this answer help you? If so, please consider:

• Adding karma to show it was useful

• Marking it as the solution if it resolved your issue

• Commenting if you need any clarification

Your feedback encourages the volunteers in this community to continue contributing.

[PickleRick]

Ok. Show us your config.

You could enable additional processing on your UF and fiddle with transforms but that would be overkill.

Setting _TCP_ROUTING should be enough but you must do it properly.

Check with 

splunk list monitor

Which inputs correspond to the data you want to ingest and overwrite the routes in those stanzas.

Alternatively you could just set everything by default to the output group which normally receives the _internal data and only overwrite routing on wineventlog inputs.

EDIT: Oh, and [splunkd] is not an input stanza but a props.conf stanza so you can't assign routing there directly.