topic Re: filtering data before indexer in Getting Data In

filtering data before indexer

abrice — Tue, 29 Sep 2020 20:43:19 GMT

I'm trying to filter data before it reaches the indexers ON the index server (I'm using universal forwarders...). I've tested on regex101 and verified the regex is good. I'm still seeing data coming in from the 4 IP's.

I have this setup:

props.conf
[Filter_Logs]
TRANSFORMS-null = null,Filter_Logs

transforms.conf
[null]
REGEX = .
DEST_KEY = queue
FORMAT = indexQueue

[Filter_Logs]
REGEX = (10.1.1.1 -|10.1.1.2 -|10.2.1.1 - |10.2.1.2 -)
DEST_KEY = queue
FORMAT = nullQueue

Re: filtering data before indexer

pruthvikrishnap — Fri, 03 Aug 2018 19:00:35 GMT

Hi Abrice,

UF do not have the capability which you are looking for, you need to replace your current universal forwarder with a heavy forwarder. Can copy all of your configuration files from the UF to the HF without having to rewrite anything.
Below article helps you configure.

http://docs.splunk.com/Documentation/Splunk/6.2.2/Forwarding/Routeandfilterdatad
Let me know if this helps.

Re: filtering data before indexer

abrice — Fri, 03 Aug 2018 19:06:34 GMT

Thank you for the reply.

I'm trying to put the filtering on the indexers and not the forwarders as we already have the forwarders deployed and don't want to change that. From what I understand the filtering can be done in either location, either at the forwarders or on the index servers.

Re: filtering data before indexer

sudosplunk — Fri, 03 Aug 2018 19:35:32 GMT

IMO, these configurations should be pushed to both UF and Indexer(s). I've seen filtering not working properly if configs are not present on both source and destination. Worth trying!

Re: filtering data before indexer

abrice — Fri, 03 Aug 2018 19:54:39 GMT

Just tried that on one web server, no luck.

on the indexers I've tried it via pushing from the cluster master as an app as well as putting it in etc/system/local

Re: filtering data before indexer

sudosplunk — Thu, 09 Aug 2018 15:54:31 GMT

Hello, were you able to solve this? If you sense everything is correct, then please revisit your regex again. In your regex (the one in question), I noticed there are whitespace characters which seems to be little off. Is it possible to provide some sample events?