Filtering on UF for Specific Events then Delete the Rest (6.3.2)

[markschoonover]

Hello Splunkers,

I've been working on filtering IIS events. What I need to keep is any event that contains auth.owa, then nullQueue the rest. I've been through the docs many times but something is missing. I've been able to send all events to nullQueue no problem but I've not been able to let the wanted events through.

This is the message I'm interested in:

2016-03-07 22:39:02 127.0.0.1 POST /OWA/auth.owa &CorrelationID=<empty>;&ClientId=AODTP0KGF0BGDQLLSW&cafeReqId=a5e157b9-047b-4e2d-a486-809dc938ed16; 443 foobar@xxx.yyy.com 127.0.0.1 AMProbe/Local/ClientAccess 200 0 0 62

Here's my props.conf file:

[source::...\\IISLogs\\...\\*]
TRANSFORMS-iis = iisbitbucket,iisauthonly

Transforms looks like this:

[iisbitbucket]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[iisauthonly]
REGEX = auth\.owa
DEST_KEY = queue
FORMAT = indexQueue

My gut is telling me I'm missing something fundamental here but I can't figure out what. I've tried all kinds of combinations of REGEX in the iisauthonly stanza but so far, nothing works. The results of this config is all messages are making it into the indexQueue. These conf files are located on the UFs which are Windows 2008 R2 servers. According to http://docs.splunk.com/Documentation/Splunk/6.2.3/Forwarding/Routeandfilterdatad#Keep_specific_event...

scroll to the bottom, structured data is filtered on the UF. Any ideas on what I'm missing?

[markschoonover · Accepted solution]

With some help from amrit on IRC, here's the config that allows filtering IIS structured data on a UF:

props.conf

[source::...\\IISLogs\\...\\*]
TRANSFORMS-iis = iisbitbucket,iisauthonly

Order of transforms stanzas is important. iisbitbucket drops everything, then iisauthonly is used to filter on the string /owa/auth.owa

transforms.conf

[iisbitbucket]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[iisauthonly]
SOURCE_KEY = field:cs_uri_stem
REGEX = (?i)/owa/auth.owa
DEST_KEY = queue
FORMAT = parsingQueue

Set the SOURCE_KEY to the field you need to filter. The /owa/auth.owa is found (as well as many other stems) in the cs_uri_stem field of the IIS structured data. REGEX is what to match in the field, DEST_KEY has to be set to queue. Some instructions about filtering on an indexer will have a FORMAT=index which is what I followed at first. After reading on all the different queues in a UF, I needed to feed the matched results back into the parsingQueue so the UF can push that to the indexer.

[amrit]

You can probably reduce to one regex/transform by using a negative lookahead:

REGEX=(?!/owa/auth.owa)

Haven't verified it, but... it should work. See more at: http://www.regular-expressions.info/lookaround.html

[markschoonover]

Regex golf amrit? 🙂

[somesoni2]

Try to have this setting on Indexer and try.

[markschoonover]

It wouldn't work on the indexer either. Since the data is structured, it's not parsed on an indexer, it goes straight into the index. Without parsing, there's no way to filter on events.