topic Re: Using Splunk heavy forwarder - Filter data before TCP routing - What's wrong with my configuration? in Getting Data In

Using Splunk heavy forwarder - Filter data before TCP routing - What's wrong with my configuration?

patouellet — Tue, 29 Sep 2020 16:20:54 GMT

Hi,

I'm using a Splunk Heavy Forwarder with props.conf, transforms.conf and outputs.conf to selectively send events to different splunk Indexers based on the sourcetype. That works well.

But now I have a new need: For a specific sourcetype (csi_pclog), I need to get rid of most events first, then route remaining events to a specific indexer. For some reason, I'm having trouble making this work. I'm not receiving any events and I've checked in the actual file I'm forwarding - there's data in there.

Basically all events for sourcetype=csi_pclog need to be dropped, except events with the word "Login". Here's what I have:

props.conf

[csi_pclog]
TRANSFORMS-routing=itseccsiappRouting, itseccsiappDiscard

transforms.conf

[itseccsiappDiscard]
REGEX=.
DEST_KEY= queue
FORMAT=nullQueue

[itseccsiappRouting]
REGEX=Login
DEST_KEY=_TCP_ROUTING
FORMAT=default-autolb-group

outputs.conf

[tcpout]
defaultGroup = default-autolb-group
forwardedindex.3.whitelist = (_audit|_internal|_introspection|_telemetry)

[tcpout:default-autolb-group]
server = indexer1:9997

[tcpout-server://indexer1:9997]

[tcpout:itsechf]
server = indexer2:9997

What am I doing wrong?

Re: Using Splunk heavy forwarder - Filter data before TCP routing - What's wrong with my configuration?

harsmarvania57 — Thu, 19 Oct 2017 17:21:06 GMT

Try to reverse order in props.conf

[csi_pclog]
TRANSFORMS-routing= itseccsiappDiscard,itseccsiappRouting

And then restart Splunk on HF

Re: Using Splunk heavy forwarder - Filter data before TCP routing - What's wrong with my configuration?

patouellet — Thu, 19 Oct 2017 19:18:10 GMT

Thanks for the reply. Tried it, no change. Still not getting events. I see the connection being established between the client and the HF, so it has to be my props/transforms/outputs combination that aren't quite right.

Re: Using Splunk heavy forwarder - Filter data before TCP routing - What's wrong with my configuration?

gjanders — Thu, 19 Oct 2017 19:36:01 GMT

If you refer to route and filter data you probably want to do (quoting from the documentation):

(1) Edit props.conf and add the following:

[source::/var/log/messages]
TRANSFORMS-set= setnull,setparsing

(2) Edit transforms.conf and add the following:
[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[setparsing]
REGEX = \[sshd\]
DEST_KEY = queue
FORMAT = indexQueue

(3) Restart Splunk Enterprise.

In other words you want to change the queue not the routing when you want to include the data you previously sent to the null queue...

Right now you do appear to be dropping everything to the null queue so no data will be sent at all.

Re: Using Splunk heavy forwarder - Filter data before TCP routing - What's wrong with my configuration?

patouellet — Thu, 19 Oct 2017 19:42:34 GMT

Thanks for the reply. I kinda get what you are saying, I think. But I'm not indexing on this HF, I want to send it to a specific tcpout group.

Let's say I do what you suggest, the events will go to the default output group, right? How to I tell it to route to a different group?

Re: Using Splunk heavy forwarder - Filter data before TCP routing - What's wrong with my configuration?

gjanders — Tue, 29 Sep 2020 16:18:07 GMT

There is likely more than one way to do this but why not specify in your inputs.conf the _TCP_ROUTING = setting? This way you can send the data to the correct TCP output.

Since the transforms.conf you have will drop everything to the null queue by default, you can use your second regular expression (Login) to ensure that data gets forwarded (and since the inputs.conf has already specified which TCP output queue you are using you won't need to override in the transforms.conf).

In terms of config:
props.conf

[csi_pclog]
TRANSFORMS-routing=itseccsiappDiscard,itseccsiappRouting

transforms.conf

[itseccsiappDiscard]
REGEX=.
DEST_KEY= queue
FORMAT=nullQueue

[itseccsiappRouting]
REGEX=Login
DEST_KEY=queue
FORMAT=indexQueue

In your inputs.conf

[<yourstanza>]
_TCP_ROUTING = itsechf

Re: Using Splunk heavy forwarder - Filter data before TCP routing - What's wrong with my configuration?

gjanders — Fri, 20 Oct 2017 10:48:16 GMT

Or do multiple transforms to change the routing and the discard/include data if you prefer...

Re: Using Splunk heavy forwarder - Filter data before TCP routing - What's wrong with my configuration?

patouellet — Fri, 20 Oct 2017 11:06:54 GMT

Think I would prefer the multiple transforms option. Care to help again? Never done it, unsure what do do.

Re: Using Splunk heavy forwarder - Filter data before TCP routing - What's wrong with my configuration?

patouellet — Fri, 27 Oct 2017 19:58:13 GMT

garethatiag answer was key to figuring this out. The full working config is:

props.conf

[csi_pclog]
TRANSFORMS-routing=itseccsiappDiscard,itseccsiappKeep, itseccsiappRouting

transforms.conf

[itseccsiappDiscard]
REGEX= .
DEST_KEY= queue
FORMAT=nullQueue

[itseccsiappKeep]
REGEX=Login
DEST_KEY=queue
FORMAT=indexQueue

[itseccsiappRouting]
REGEX= .
DEST_KEY=_TCP_ROUTING
FORMAT=itsechf