Requirment
Drop events before they get sent to the splunk indexer.
Want to just send the lines with "Authentication_failed" on the file to the indexer.
Information
Using heavy forwarder and on 4.2.4
input.conf
c:\splunk\etc\system\local\iputs.conf
[monitor://c:\Programs Files\WebProxyLogs]
sourcetype=WebProxy
props.conf
c:\splunk\etc\system\local\props.conf
[source:://c:\Programs Files\WebProxyLogs]
TRANSFORMS-set=setnull,setparsing
transforms.conf
c:\splunk\etc\system\local\transforms.conf
[setnull]
REGEX =
DEST_KEY = queue
FORMAT = nullQueue
[setparsing]
REGEX = \[Authentication_failed\]
DEST_KEY = queue
FORMAT = indexQueue
Result
Unable to index the logs
Problem
In need of guidance on how to solve this problem.
This is how I got it to work:
transforms.conf
[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue
[setparsing]
REGEX = \[(authentication failed)\]
DEST_KEY = queue
FORMAT = indexQueue
props.conf
[source:://c:\Programs Files\WebProxyLogs]
TRANSFORMS-set = setnull,setparsing
Write a regular expression that matches anything EXCEPT the authentication failed string. This is a hard regex to write, because most tools (like grep) have a "not" switch so that you don't have to put the logic into the actual regex.
But here is an attempt
REGEX=^((?!\[Authentication_failed\]).)*$
Put this REGEX in your transforms.conf [setnull]
instead of
REGEX=
In this case, props.conf could stay the same. I got this idea from http://stackoverflow.com/questions/406230/regular-expression-to-match-string-not-containing-a-word
[setnull]
REGEX =
is sending everything to the null queue. Try this:
Leave transforms.conf as it is. Change props.conf to specify the order of execution of the transforms. props.conf:
[source:://c:\Programs Files\WebProxyLogs]
TRANSFORMS-set1=setparsing
TRANSFORMS-set2=setnull
I hope that this will first send the chosen events to the index queue, and then everything else to the null queue.