How does _TCP_ROUTING work in inputs.conf?

[JarrettM]

We soon will be required to send our Windows Event Security logs to a separate Splunk sever owned by our organization's Security group. To test this, I installed a test Splunk server (testsplunk in below files). I first tested that I could send all events to both Splunk indexers. Here are are outputs.conf and inputs.conf from the Splunk Universal Forwarder client I used in this first test:

$SPLUNK_HOME/etc/system/local/outputs.conf
     [tcpout]
        defaultGroup = mysplunk, testsplunk
     [tcpout:mysplunk]
         server = mysplunk.com:9997
     [tcpout:testsplunk]
        server = testsplunk.com:9997
$SPLUNK_HOME/etc/apps/WinEvt_Logs/local/inputs.conf
    [WinEventLog://Security]
    disabled = 0
    index = winevent

In this case both servers received all events as expected (including events from 3 other apps not shown here). In the next test I wanted mysplunk to continue receive all events and testsplunk to only get [WinEventLog://Security] To accomplish this I took testsplunk out of the defaultGroup and modified inputs.conf as shown below:

$SPLUNK_HOME/etc/system/local/outputs.conf
     [tcpout]
        defaultGroup = mysplunk
     [tcpout:mysplunk]
        server = mysplunk.com:9997
     [tcpout:testsplunk]
        server = testsplunk.com:9997
$SPLUNK_HOME/etc/apps/WinEvt_Logs/local/inputs.conf
    [WinEventLog://Security]
    _TCP_ROUTING = mysplunk, testsplunk
    disabled = 0
    index = winevent

After restarting the SplunkForwarder, mysplunk did keep receiving all events but testsplunk now got nothing.
What am I missing?

[pruthvikrishnap]

Hi,

Try adding testsplunk to the default group, else exclude the default group.
This will enable [WinEventLog://Security] logs to both mysplunk and testsplunk

[JarrettM]

Thanks for your reply. But here's what I've found. I've tried all different combinations of _TCP_ROUTING and defaultGroup. If a server is in the defaultGroup it gets all data from all the apps. If it's not it gets nothing. _TCP_routing has no effect no matter which server is in it. The statement seems to be completely ignored

[sudosplunk]

Get rid of mysplunk group from _TCP_ROUTING and this should route security event logs to testsplunk and other data to mysplunk.

[WinEventLog://Security]
_TCP_ROUTING = testsplunk
disabled = 0
index = winevent

Your outputs.conf looks good.

[JarrettM]

Thanks for your reply. But here's what I've found. Taking your suggestion I've tried all different combinations of _TCP_ROUTING and defaultGroup. If a server is in the defaultGroup it gets all data from all the apps. If it's not it gets nothing. _TCP_routing has no effect no matter which server is in it. The statement seems to be completely ignored.

[JLewis21]

Did you ever figure out a fix for this issue? I am seeing the exact same behavior for our windows inputs.

[joesrepsolc]

it is case sensitive... so ALL CAPS on that _TCP_ROUTING part.

[sudosplunk]

I've tested this and was able to route events as desired.
_TCP_ROUTING=testsplunk will (and should) route data (from respective monitor stanza) to 'testsplunk' group.
defaultGroup=mysplunk (in ouputs.conf) will route all other data (including _internal logs) to 'mysplunk' group. If this is not working, then please check for typos and see that there isn't any precedence issue. On forwarder, run this command to check various inputs.conf settings which splunk is taking into consideration.
From $SPLUNK_HOME/bin/, splunk btool inputs list --debug

Just reminding, forwarder must be restarted for the changes to take effect. More information about, Route inputs to specific indexers based on the data input

[JarrettM]

Thanks. I obviously have a mis-configuration somewhere. I'll keep hacking at it and let you know when I figure it out.