Pre-index filtering

[leiniao]

Requirment

Drop events before they get sent to the splunk indexer.

Want to just send the lines with "Authentication_failed" on the file to the indexer.

Information

Using heavy forwarder and on 4.2.4

input.conf

c:\splunk\etc\system\local\iputs.conf

[monitor://c:\Programs Files\WebProxyLogs]
sourcetype=WebProxy

props.conf

c:\splunk\etc\system\local\props.conf

[source:://c:\Programs Files\WebProxyLogs] 
TRANSFORMS-set=setnull,setparsing

transforms.conf

c:\splunk\etc\system\local\transforms.conf

[setnull] 
REGEX =
DEST_KEY = queue
FORMAT = nullQueue

[setparsing] 
REGEX = \[Authentication_failed\]
DEST_KEY = queue
FORMAT = indexQueue

Result

Unable to index the logs

Problem
In need of guidance on how to solve this problem.

[twkan]

This is how I got it to work:

transforms.conf

[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[setparsing]
REGEX = \[(authentication failed)\]
DEST_KEY = queue
FORMAT = indexQueue

props.conf

 [source:://c:\Programs Files\WebProxyLogs] 
TRANSFORMS-set = setnull,setparsing

[lguinn2]

Write a regular expression that matches anything EXCEPT the authentication failed string. This is a hard regex to write, because most tools (like grep) have a "not" switch so that you don't have to put the logic into the actual regex.

But here is an attempt

REGEX=^((?!\[Authentication_failed\]).)*$

Put this REGEX in your transforms.conf [setnull] instead of

REGEX=

In this case, props.conf could stay the same. I got this idea from http://stackoverflow.com/questions/406230/regular-expression-to-match-string-not-containing-a-word

[lguinn2]

[setnull] 
REGEX =

is sending everything to the null queue. Try this:

Leave transforms.conf as it is. Change props.conf to specify the order of execution of the transforms. props.conf:

[source:://c:\Programs Files\WebProxyLogs] 
TRANSFORMS-set1=setparsing
TRANSFORMS-set2=setnull

I hope that this will first send the chosen events to the index queue, and then everything else to the null queue.