topic Re: Pre-index filtering in Getting Data In

Pre-index filtering

leiniao — Wed, 18 Jan 2012 08:25:41 GMT

Requirment

Drop events before they get sent to the splunk indexer.

Want to just send the lines with "Authentication_failed" on the file to the indexer.

Information

Using heavy forwarder and on 4.2.4

input.conf

c:\splunk\etc\system\local\iputs.conf

[monitor://c:\Programs Files\WebProxyLogs]
sourcetype=WebProxy

props.conf

c:\splunk\etc\system\local\props.conf

[source:://c:\Programs Files\WebProxyLogs] 
TRANSFORMS-set=setnull,setparsing

transforms.conf

c:\splunk\etc\system\local\transforms.conf

[setnull] 
REGEX =
DEST_KEY = queue
FORMAT = nullQueue

[setparsing] 
REGEX = \[Authentication_failed\]
DEST_KEY = queue
FORMAT = indexQueue

Result

Unable to index the logs

Problem
In need of guidance on how to solve this problem.

Re: Pre-index filtering

lguinn2 — Wed, 18 Jan 2012 08:42:10 GMT

[setnull] 
REGEX =

is sending everything to the null queue. Try this:

Leave transforms.conf as it is. Change props.conf to specify the order of execution of the transforms. props.conf:

[source:://c:\Programs Files\WebProxyLogs] 
TRANSFORMS-set1=setparsing
TRANSFORMS-set2=setnull

I hope that this will first send the chosen events to the index queue, and then everything else to the null queue.

Re: Pre-index filtering

lguinn2 — Wed, 18 Jan 2012 08:54:50 GMT

Write a regular expression that matches anything EXCEPT the authentication failed string. This is a hard regex to write, because most tools (like grep) have a "not" switch so that you don't have to put the logic into the actual regex.

But here is an attempt

REGEX=^((?!\[Authentication_failed\]).)*$

Put this REGEX in your transforms.conf [setnull] instead of

REGEX=

In this case, props.conf could stay the same. I got this idea from http://stackoverflow.com/questions/406230/regular-expression-to-match-string-not-containing-a-word

Re: Pre-index filtering

twkan — Fri, 17 Feb 2012 19:36:59 GMT

This is how I got it to work:

transforms.conf

[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[setparsing]
REGEX = \[(authentication failed)\]
DEST_KEY = queue
FORMAT = indexQueue

props.conf

 [source:://c:\Programs Files\WebProxyLogs] 
TRANSFORMS-set = setnull,setparsing