All Apps and Add-ons

_TCP_ROUTING for Heavy Forwarder

Motivator

Hi Splunkers,

I've a situation where TCPROUTING setting in inputs.conf is not being honored by splunk. Here is my architecture and related config files.

HF --> Indexer cluster

On HF, $SPLUNKHOME/etc/apps/SplunkTAcheckpoint/local/opsecleainputs.conf:

[FirewallEvents]
_TCP_ROUTING = fw_cluster

On HF, $SPLUNKHOME/etc/apps/routeoutputs/local/outpus.conf:

[tcpout]
indexAndForward = false
defaultGroup = main_cluster
autoLBFrequency = 15

[tcpout:main_cluster]
server = mainIDX1:9997,mainIDX2:9997,mainIDX3:9997,mainIDX4:9997
useACK = true
maxQueueSize = 7MB

[tcpout:fw_cluster]
server = fwIDX1:9997,fwIDX2:9997,fwIDX3:9997,fwIDX4:9997
useACK = true
maxQueueSize = 7MB

Events are still being routed to main_cluster instead of fw_cluster. This kind of routing is working for other data sources coming through UFs.

I've already reviewed metrics.log and splunkd.log and validated HF is making TCPInput connections to indexers (fw_cluster).

Any advise on troubleshooting is appreciated.

0 Karma
1 Solution

Path Finder

Setting meta fields via inputs.conf does not work for modular inputs. Because modular inputs insert data after the typing queue, it's not possible to use inputs.conf or props/transforms.conf overrides to change meta fields. Modular inputs will always go to your default output group.

As an aside, handling routing for any type of data is something our product does, if you want to check it out at https://cribl.io/. Fits natively into a heavyweight forwarder or indexer.

View solution in original post

Contributor

Hi,

Routing with pros.conf and transforms.conf instead of inputs.conf should do the job:

props:

[opsec]

TRANSFORMS-route=routefw_cluster

transforms:

[routefw_cluster]

REGEX=.+
DEST_KEY=_TCP_ROUTING
FORMAT=fw_cluster

Best regards,

Andreas

0 Karma

Path Finder

Setting meta fields via inputs.conf does not work for modular inputs. Because modular inputs insert data after the typing queue, it's not possible to use inputs.conf or props/transforms.conf overrides to change meta fields. Modular inputs will always go to your default output group.

As an aside, handling routing for any type of data is something our product does, if you want to check it out at https://cribl.io/. Fits natively into a heavyweight forwarder or indexer.

View solution in original post

Motivator

Hi @coccyx,

I had a doubt that this has something to do with add-ons since similar kind of implementation works for other inputs (from UFs, syslog) but not for these add-ons. Thanks for the answer. I will have to look into "https://cribl.io" and will discuss with my team.

0 Karma

Ultra Champion

What is _TCP_ROUTING for this specific input when running ./splunk cmd btool inputs list --debug?

0 Karma

Motivator

Hi @ddrillic, splunk is able to recognize the setting from input.conf of only Splunk_TA_box but not Splunk_TA_checkpoint.

[splunk@HF ~]$ /splunk/bin/splunk btool inputs list --debug | grep '_TCP_ROUTING'
/splunk/etc/apps/Splunk_TA_box/local/inputs.conf                   _TCP_ROUTING = fw_cluster
/splunk/etc/system/default/inputs.conf                             _TCP_ROUTING = *
0 Karma

Ultra Champion

Anything that ./splunk cmd btool check reports? maybe a syntax error...

Motivator

Hi @ddrillic ,

Below is the output. Based on this output and @coccyx answer, I believe that _TCP_ROUTING setting doesn't work for modular inputs.

[splunk@HF local]$ /splunk/bin/splunk cmd btool check
                Invalid key in stanza [FirewallEvents] in /splunk/etc/apps/Splunk_TA_checkpoint-opseclea/local/opseclea_inputs.conf, line 2: _TCP_ROUTING  (value:  fw_cluster).
                Invalid key in stanza [FirewallAudit] in /splunk/etc/apps/Splunk_TA_checkpoint-opseclea/local/opseclea_inputs.conf, line 12: _TCP_ROUTING  (value:  fw_cluster).
[splunk@pespl027 local]$
0 Karma

Ultra Champion

It's been a while since I've worked with the opsec lea addon, but doesn't that generate multiple input stanzas, one for each log collection you define in the opsec add-on GUI? So shouldn't you be adding that TCProuting setting in each of those generated input.conf sections?

0 Karma

Motivator

Yes it does. I have 2 stanzas one for FirewallEvents and one for FirewallAudit. I added TCPROUTING under both the stanzas in inputs.conf but did not specify that in the question.

Another interesting thing: We are using SplunkTAbox on HF and these events are also not routed to fwcluster even after explicitly defining _TCPROUTING under stanza.

0 Karma