All Apps and Add-ons

_TCP_ROUTING for Heavy Forwarder

sudosplunk
Motivator

Hi Splunkers,

I've a situation where _TCP_ROUTING setting in inputs.conf is not being honored by splunk. Here is my architecture and related config files.

HF --> Indexer cluster

On HF, $SPLUNK_HOME/etc/apps/Splunk_TA_checkpoint/local/opseclea_inputs.conf:

[FirewallEvents]
_TCP_ROUTING = fw_cluster

On HF, $SPLUNK_HOME/etc/apps/route_outputs/local/outpus.conf:

[tcpout]
indexAndForward = false
defaultGroup = main_cluster
autoLBFrequency = 15

[tcpout:main_cluster]
server = mainIDX1:9997,mainIDX2:9997,mainIDX3:9997,mainIDX4:9997
useACK = true
maxQueueSize = 7MB

[tcpout:fw_cluster]
server = fwIDX1:9997,fwIDX2:9997,fwIDX3:9997,fwIDX4:9997
useACK = true
maxQueueSize = 7MB

Events are still being routed to main_cluster instead of fw_cluster. This kind of routing is working for other data sources coming through UFs.

I've already reviewed metrics.log and splunkd.log and validated HF is making TCPInput connections to indexers (fw_cluster).

Any advise on troubleshooting is appreciated.

0 Karma
1 Solution

coccyx
Path Finder

Setting meta fields via inputs.conf does not work for modular inputs. Because modular inputs insert data after the typing queue, it's not possible to use inputs.conf or props/transforms.conf overrides to change meta fields. Modular inputs will always go to your default output group.

As an aside, handling routing for any type of data is something our product does, if you want to check it out at https://cribl.io/. Fits natively into a heavyweight forwarder or indexer.

View solution in original post

schose
Builder

Hi,

Routing with pros.conf and transforms.conf instead of inputs.conf should do the job:

props:

[opsec]

TRANSFORMS-route=routefw_cluster

transforms:

[routefw_cluster]

REGEX=.+
DEST_KEY=_TCP_ROUTING
FORMAT=fw_cluster

Best regards,

Andreas

0 Karma

coccyx
Path Finder

Setting meta fields via inputs.conf does not work for modular inputs. Because modular inputs insert data after the typing queue, it's not possible to use inputs.conf or props/transforms.conf overrides to change meta fields. Modular inputs will always go to your default output group.

As an aside, handling routing for any type of data is something our product does, if you want to check it out at https://cribl.io/. Fits natively into a heavyweight forwarder or indexer.

View solution in original post

jianw223
Loves-to-Learn

This is an endorsement by a Cribl employee. As a previous user of Cribl, I would not recommend it. It is considerably slower and buggy.

0 Karma

sudosplunk
Motivator

Hi @coccyx,

I had a doubt that this has something to do with add-ons since similar kind of implementation works for other inputs (from UFs, syslog) but not for these add-ons. Thanks for the answer. I will have to look into "https://cribl.io" and will discuss with my team.

ddrillic
Ultra Champion

What is _TCP_ROUTING for this specific input when running ./splunk cmd btool inputs list --debug?

0 Karma

sudosplunk
Motivator

Hi @ddrillic, splunk is able to recognize the setting from input.conf of only Splunk_TA_box but not Splunk_TA_checkpoint.

[splunk@HF ~]$ /splunk/bin/splunk btool inputs list --debug | grep '_TCP_ROUTING'
/splunk/etc/apps/Splunk_TA_box/local/inputs.conf                   _TCP_ROUTING = fw_cluster
/splunk/etc/system/default/inputs.conf                             _TCP_ROUTING = *
0 Karma

ddrillic
Ultra Champion

Anything that ./splunk cmd btool check reports? maybe a syntax error...

sudosplunk
Motivator

Hi @ddrillic ,

Below is the output. Based on this output and @coccyx answer, I believe that _TCP_ROUTING setting doesn't work for modular inputs.

[splunk@HF local]$ /splunk/bin/splunk cmd btool check
                Invalid key in stanza [FirewallEvents] in /splunk/etc/apps/Splunk_TA_checkpoint-opseclea/local/opseclea_inputs.conf, line 2: _TCP_ROUTING  (value:  fw_cluster).
                Invalid key in stanza [FirewallAudit] in /splunk/etc/apps/Splunk_TA_checkpoint-opseclea/local/opseclea_inputs.conf, line 12: _TCP_ROUTING  (value:  fw_cluster).
[splunk@pespl027 local]$
0 Karma

FrankVl
Ultra Champion

It's been a while since I've worked with the opsec lea addon, but doesn't that generate multiple input stanzas, one for each log collection you define in the opsec add-on GUI? So shouldn't you be adding that _TCP_routing setting in each of those generated input.conf sections?

0 Karma

sudosplunk
Motivator

Yes it does. I have 2 stanzas one for FirewallEvents and one for FirewallAudit. I added _TCP_ROUTING under both the stanzas in inputs.conf but did not specify that in the question.

Another interesting thing: We are using Splunk_TA_box on HF and these events are also not routed to fw_cluster even after explicitly defining _TCP_ROUTING under stanza.

0 Karma
Did you miss .conf21 Virtual?

Good news! The event's keynotes and many of its breakout sessions are now available online, and still totally FREE!