Archive

filtering data before indexer

Explorer

I'm trying to filter data before it reaches the indexers ON the index server (I'm using universal forwarders...). I've tested on regex101 and verified the regex is good. I'm still seeing data coming in from the 4 IP's.

I have this setup:

props.conf
[Filter_Logs]
TRANSFORMS-null = null,Filter_Logs

transforms.conf
[null]
REGEX = .
DEST_KEY = queue
FORMAT = indexQueue

[Filter_Logs]
REGEX = (10.1.1.1 -|10.1.1.2 -|10.2.1.1 - |10.2.1.2 -)
DEST_KEY = queue
FORMAT = nullQueue

Tags (1)
0 Karma

Explorer

Thank you for the reply.

I'm trying to put the filtering on the indexers and not the forwarders as we already have the forwarders deployed and don't want to change that. From what I understand the filtering can be done in either location, either at the forwarders or on the index servers.

0 Karma

Motivator

IMO, these configurations should be pushed to both UF and Indexer(s). I've seen filtering not working properly if configs are not present on both source and destination. Worth trying!

0 Karma

Explorer

Just tried that on one web server, no luck.

on the indexers I've tried it via pushing from the cluster master as an app as well as putting it in etc/system/local

0 Karma

Motivator

Hello, were you able to solve this? If you sense everything is correct, then please revisit your regex again. In your regex (the one in question), I noticed there are whitespace characters which seems to be little off. Is it possible to provide some sample events?

0 Karma

Contributor

Hi Abrice,

UF do not have the capability which you are looking for, you need to replace your current universal forwarder with a heavy forwarder. Can copy all of your configuration files from the UF to the HF without having to rewrite anything.
Below article helps you configure.

http://docs.splunk.com/Documentation/Splunk/6.2.2/Forwarding/Routeandfilterdatad
Let me know if this helps.

0 Karma
State of Splunk Careers

Access the Splunk Careers Report to see real data that shows how Splunk mastery increases your value and job satisfaction.

Find out what your skills are worth!