_TCP_ROUTING for Heavy Forwarder

[sudosplunk]

Hi Splunkers,

I've a situation where _TCP_ROUTING setting in inputs.conf is not being honored by splunk. Here is my architecture and related config files.

HF --> Indexer cluster

On HF, $SPLUNK_HOME/etc/apps/Splunk_TA_checkpoint/local/opseclea_inputs.conf:

[FirewallEvents]
_TCP_ROUTING = fw_cluster

On HF, $SPLUNK_HOME/etc/apps/route_outputs/local/outpus.conf:

[tcpout]
indexAndForward = false
defaultGroup = main_cluster
autoLBFrequency = 15

[tcpout:main_cluster]
server = mainIDX1:9997,mainIDX2:9997,mainIDX3:9997,mainIDX4:9997
useACK = true
maxQueueSize = 7MB

[tcpout:fw_cluster]
server = fwIDX1:9997,fwIDX2:9997,fwIDX3:9997,fwIDX4:9997
useACK = true
maxQueueSize = 7MB

Events are still being routed to main_cluster instead of fw_cluster. This kind of routing is working for other data sources coming through UFs.

I've already reviewed metrics.log and splunkd.log and validated HF is making TCPInput connections to indexers (fw_cluster).

Any advise on troubleshooting is appreciated.

[hrawat_splunk]

@sudosplunk the setting is expected to be in inputs.conf ( not in custom inputs.conf).
All modinputs honor meta setting starting 6.4
https://community.splunk.com/t5/Getting-Data-In/UF-Route-inputs-to-specific-indexers-based-on-the-da...

[schose]

Hi,

Routing with pros.conf and transforms.conf instead of inputs.conf should do the job:

props:

[opsec]

TRANSFORMS-route=routefw_cluster

transforms:

[routefw_cluster]

REGEX=.+
DEST_KEY=_TCP_ROUTING
FORMAT=fw_cluster

Best regards,

Andreas

[ddrillic]

What is _TCP_ROUTING for this specific input when running ./splunk cmd btool inputs list --debug?

[sudosplunk]

Hi @ddrillic, splunk is able to recognize the setting from input.conf of only Splunk_TA_box but not Splunk_TA_checkpoint.

[splunk@HF ~]$ /splunk/bin/splunk btool inputs list --debug | grep '_TCP_ROUTING'
/splunk/etc/apps/Splunk_TA_box/local/inputs.conf                   _TCP_ROUTING = fw_cluster
/splunk/etc/system/default/inputs.conf                             _TCP_ROUTING = *

[ddrillic]

Anything that ./splunk cmd btool check reports? maybe a syntax error...

[sudosplunk]

Hi @ddrillic ,

Below is the output. Based on this output and @coccyx answer, I believe that _TCP_ROUTING setting doesn't work for modular inputs.

[splunk@HF local]$ /splunk/bin/splunk cmd btool check
                Invalid key in stanza [FirewallEvents] in /splunk/etc/apps/Splunk_TA_checkpoint-opseclea/local/opseclea_inputs.conf, line 2: _TCP_ROUTING  (value:  fw_cluster).
                Invalid key in stanza [FirewallAudit] in /splunk/etc/apps/Splunk_TA_checkpoint-opseclea/local/opseclea_inputs.conf, line 12: _TCP_ROUTING  (value:  fw_cluster).
[splunk@pespl027 local]$

[FrankVl]

It's been a while since I've worked with the opsec lea addon, but doesn't that generate multiple input stanzas, one for each log collection you define in the opsec add-on GUI? So shouldn't you be adding that _TCP_routing setting in each of those generated input.conf sections?

[sudosplunk]

Yes it does. I have 2 stanzas one for FirewallEvents and one for FirewallAudit. I added _TCP_ROUTING under both the stanzas in inputs.conf but did not specify that in the question.

Another interesting thing: We are using Splunk_TA_box on HF and these events are also not routed to fw_cluster even after explicitly defining _TCP_ROUTING under stanza.