Hi Splunkers,
I've a situation where _TCP_ROUTING setting in inputs.conf is not being honored by splunk. Here is my architecture and related config files.
HF --> Indexer cluster
On HF, $SPLUNK_HOME/etc/apps/Splunk_TA_checkpoint/local/opseclea_inputs.conf:
[FirewallEvents]
_TCP_ROUTING = fw_cluster
On HF, $SPLUNK_HOME/etc/apps/route_outputs/local/outpus.conf:
[tcpout]
indexAndForward = false
defaultGroup = main_cluster
autoLBFrequency = 15
[tcpout:main_cluster]
server = mainIDX1:9997,mainIDX2:9997,mainIDX3:9997,mainIDX4:9997
useACK = true
maxQueueSize = 7MB
[tcpout:fw_cluster]
server = fwIDX1:9997,fwIDX2:9997,fwIDX3:9997,fwIDX4:9997
useACK = true
maxQueueSize = 7MB
Events are still being routed to main_cluster
instead of fw_cluster
. This kind of routing is working for other data sources coming through UFs.
I've already reviewed metrics.log and splunkd.log and validated HF is making TCPInput connections to indexers (fw_cluster).
Any advise on troubleshooting is appreciated.
Setting meta fields via inputs.conf does not work for modular inputs. Because modular inputs insert data after the typing queue, it's not possible to use inputs.conf or props/transforms.conf overrides to change meta fields. Modular inputs will always go to your default output group.
As an aside, handling routing for any type of data is something our product does, if you want to check it out at https://cribl.io/. Fits natively into a heavyweight forwarder or indexer.
Hi,
Routing with pros.conf and transforms.conf instead of inputs.conf should do the job:
props:
[opsec]
TRANSFORMS-route=routefw_cluster
transforms:
[routefw_cluster]
REGEX=.+
DEST_KEY=_TCP_ROUTING
FORMAT=fw_cluster
Best regards,
Andreas
Setting meta fields via inputs.conf does not work for modular inputs. Because modular inputs insert data after the typing queue, it's not possible to use inputs.conf or props/transforms.conf overrides to change meta fields. Modular inputs will always go to your default output group.
As an aside, handling routing for any type of data is something our product does, if you want to check it out at https://cribl.io/. Fits natively into a heavyweight forwarder or indexer.
This is an endorsement by a Cribl employee. As a previous user of Cribl, I would not recommend it. It is considerably slower and buggy.
Hi @coccyx,
I had a doubt that this has something to do with add-ons since similar kind of implementation works for other inputs (from UFs, syslog) but not for these add-ons. Thanks for the answer. I will have to look into "https://cribl.io" and will discuss with my team.
What is _TCP_ROUTING
for this specific input when running ./splunk cmd btool inputs list --debug
?
Hi @ddrillic, splunk is able to recognize the setting from input.conf of only Splunk_TA_box
but not Splunk_TA_checkpoint
.
[splunk@HF ~]$ /splunk/bin/splunk btool inputs list --debug | grep '_TCP_ROUTING'
/splunk/etc/apps/Splunk_TA_box/local/inputs.conf _TCP_ROUTING = fw_cluster
/splunk/etc/system/default/inputs.conf _TCP_ROUTING = *
Anything that ./splunk cmd btool check
reports? maybe a syntax error...
Hi @ddrillic ,
Below is the output. Based on this output and @coccyx answer, I believe that _TCP_ROUTING
setting doesn't work for modular inputs.
[splunk@HF local]$ /splunk/bin/splunk cmd btool check
Invalid key in stanza [FirewallEvents] in /splunk/etc/apps/Splunk_TA_checkpoint-opseclea/local/opseclea_inputs.conf, line 2: _TCP_ROUTING (value: fw_cluster).
Invalid key in stanza [FirewallAudit] in /splunk/etc/apps/Splunk_TA_checkpoint-opseclea/local/opseclea_inputs.conf, line 12: _TCP_ROUTING (value: fw_cluster).
[splunk@pespl027 local]$
It's been a while since I've worked with the opsec lea addon, but doesn't that generate multiple input stanzas, one for each log collection you define in the opsec add-on GUI? So shouldn't you be adding that _TCP_routing setting in each of those generated input.conf sections?
Yes it does. I have 2 stanzas one for FirewallEvents and one for FirewallAudit. I added _TCP_ROUTING under both the stanzas in inputs.conf but did not specify that in the question.
Another interesting thing: We are using Splunk_TA_box on HF and these events are also not routed to fw_cluster even after explicitly defining _TCP_ROUTING under stanza.