Getting Data In

_internal log tcp routing

TheBravoSierra
Path Finder

Splunk Universal Forwarder - Route wineventlog to one output group and _internal to another

I'm trying to configure a Universal Forwarder so that:

  • wineventlog data is sent only to [tcpout:group1]
  • _internal data is sent only to [tcpout:group2]

Currently, wineventlog is routing correctly to group1, but _internal events are being sent to both group1 and group2.

My goal is to stop _internal events from being forwarded to group1 and have them forwarded exclusively to group2.

I've already tried overriding _TCP_ROUTING=* by setting _TCP_ROUTING = group2 on both the monitor://...splunkd.log stanza and the [splunkd] stanza, but _internal data is still being forwarded to both output groups.

I've also reviewed Splunk's routing and filtering documentation, but I'm not sure what additional configuration is needed.

What is the correct way to exclude _internal events from group1 while continuing to send them to group2? Is there a props/transforms-based routing configuration or another setting I'm overlooking?

Labels (1)
0 Karma

Happy_16
Observer

This isn't your config — it's a default the UF ships with. The internal-log monitor stanzas in $SPLUNK_HOME/etc/apps/SplunkUniversalForwarder/default/inputs.conf come with _TCP_ROUTING = *, which means "send to all tcpout groups." So the moment you added group2, _internal started cloning to both.

Your override was close but missed two things:

  • [splunkd] is a sourcetype, not an input stanza — _TCP_ROUTING only works on the actual [monitor://...] stanza, so that line does nothing.
  • _internal is fed by multiple monitors (splunkd.log, metrics.log, etc.), each with its own _TCP_ROUTING = *. You only overrode splunkd.log; the rest keep cloning to both groups.

(Side note: props/transforms routing won't work on a UF — it doesn't parse. _TCP_ROUTING in inputs.conf is the only lever. And internal indexes require an explicit _TCP_ROUTING, so override it to group2 — don't delete it.)

Fix. Enumerate what your version actually ships:

 
$SPLUNK_HOME/bin/splunk btool inputs list --debug | grep -i "TCP_ROUTING"

Then in $SPLUNK_HOME/etc/system/local/inputs.conf, override each internal stanza (header + routing line is enough; index/sourcetype still layer in from default):

 
[monitor://$SPLUNK_HOME/var/log/splunk/splunkd.log]
_TCP_ROUTING = group2

[monitor://$SPLUNK_HOME/var/log/splunk/metrics.log]
_TCP_ROUTING = group2

Add a block for every _internal stanza btool listed — any you skip keeps going to group1. Same applies to _audit (audit.log) and _introspection if you want those off group1 too. Restart, then re-run the btool grep to confirm they all resolve to group2.


If you meant something other than a reply to post, let me know.

0 Karma

livehybrid
SplunkTrust
SplunkTrust

Hi @TheBravoSierra 

Im not convinced that adding a monitor://...splunkd.log stanza would overwrite the default "monitor://$SPLUNK_HOME/var/log/splunk" which picks up these files usually.

Could you try with the following?

[monitor://$SPLUNK_HOME/var/log/splunk]
_TCP_ROUTING = group2

🌟 Did this answer help you? If so, please consider:

    • Adding karma to show it was useful
    • Marking it as the solution if it resolved your issue
    • Commenting if you need any clarification

Your feedback encourages the volunteers in this community to continue contributing.

0 Karma

PickleRick
SplunkTrust
SplunkTrust

Ok. Show us your config.

You could enable additional processing on your UF and fiddle with transforms but that would be overkill.

Setting _TCP_ROUTING should be enough but you must do it properly.

Check with 

splunk list monitor

Which inputs correspond to the data you want to ingest and overwrite the routes in those stanzas.

Alternatively you could just set everything by default to the output group which normally receives the _internal data and only overwrite routing on wineventlog inputs.

EDIT: Oh, and [splunkd] is not an input stanza but a props.conf stanza so you can't assign routing there directly.

0 Karma
Got questions? Get answers!

Join the Splunk Community Slack to learn, troubleshoot, and make connections with fellow Splunk practitioners in real time!

Meet up IRL or virtually!

Join Splunk User Groups to connect and learn in-person by region or remotely by topic or industry.

Get Updates on the Splunk Community!

Federated Search for Snowflake Is Now Generally Available on Splunk Cloud Platform

Splunk is excited to announce the General Availability (GA) of Federated Search for ...

Help Us Build Better Splunk Regex Puzzles (And Win Prizes!)

If you’ve spent any time in the Splunk Community Slack, you’ve likely seen our resident Splunk Trust ...

Fuel Your Journey: What’s Waiting for You at the .conf26 Acceleration Station

Navigating the show floor at .conf26 isn't just about keynotes and technical breakout sessions; it's also ...