Archive

Filtering on UF for Specific Events then Delete the Rest (6.3.2)

Explorer

Hello Splunkers,

I've been working on filtering IIS events. What I need to keep is any event that contains auth.owa, then nullQueue the rest. I've been through the docs many times but something is missing. I've been able to send all events to nullQueue no problem but I've not been able to let the wanted events through.

This is the message I'm interested in:

2016-03-07 22:39:02 127.0.0.1 POST /OWA/auth.owa &CorrelationID=<empty>;&ClientId=AODTP0KGF0BGDQLLSW&cafeReqId=a5e157b9-047b-4e2d-a486-809dc938ed16; 443 foobar@xxx.yyy.com 127.0.0.1 AMProbe/Local/ClientAccess 200 0 0 62

Here's my props.conf file:

[source::...\\IISLogs\\...\\*]
TRANSFORMS-iis = iisbitbucket,iisauthonly

Transforms looks like this:

[iisbitbucket]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[iisauthonly]
REGEX = auth\.owa
DEST_KEY = queue
FORMAT = indexQueue

My gut is telling me I'm missing something fundamental here but I can't figure out what. I've tried all kinds of combinations of REGEX in the iisauthonly stanza but so far, nothing works. The results of this config is all messages are making it into the indexQueue. These conf files are located on the UFs which are Windows 2008 R2 servers. According to http://docs.splunk.com/Documentation/Splunk/6.2.3/Forwarding/Routeandfilterdatad#Keep_specific_event...

scroll to the bottom, structured data is filtered on the UF. Any ideas on what I'm missing?

1 Solution

Explorer

With some help from amrit on IRC, here's the config that allows filtering IIS structured data on a UF:

props.conf

[source::...\\IISLogs\\...\\*]
TRANSFORMS-iis = iisbitbucket,iisauthonly

Order of transforms stanzas is important. iisbitbucket drops everything, then iisauthonly is used to filter on the string /owa/auth.owa

transforms.conf

[iisbitbucket]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[iisauthonly]
SOURCE_KEY = field:cs_uri_stem
REGEX = (?i)/owa/auth.owa
DEST_KEY = queue
FORMAT = parsingQueue

Set the SOURCEKEY to the field you need to filter. The /owa/auth.owa is found (as well as many other stems) in the csuristem field of the IIS structured data. REGEX is what to match in the field, DESTKEY has to be set to queue. Some instructions about filtering on an indexer will have a FORMAT=index which is what I followed at first. After reading on all the different queues in a UF, I needed to feed the matched results back into the parsingQueue so the UF can push that to the indexer.

View solution in original post

Explorer

With some help from amrit on IRC, here's the config that allows filtering IIS structured data on a UF:

props.conf

[source::...\\IISLogs\\...\\*]
TRANSFORMS-iis = iisbitbucket,iisauthonly

Order of transforms stanzas is important. iisbitbucket drops everything, then iisauthonly is used to filter on the string /owa/auth.owa

transforms.conf

[iisbitbucket]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[iisauthonly]
SOURCE_KEY = field:cs_uri_stem
REGEX = (?i)/owa/auth.owa
DEST_KEY = queue
FORMAT = parsingQueue

Set the SOURCEKEY to the field you need to filter. The /owa/auth.owa is found (as well as many other stems) in the csuristem field of the IIS structured data. REGEX is what to match in the field, DESTKEY has to be set to queue. Some instructions about filtering on an indexer will have a FORMAT=index which is what I followed at first. After reading on all the different queues in a UF, I needed to feed the matched results back into the parsingQueue so the UF can push that to the indexer.

View solution in original post

Splunk Employee
Splunk Employee

You can probably reduce to one regex/transform by using a negative lookahead:

REGEX=(?!/owa/auth.owa)

Haven't verified it, but... it should work. See more at: http://www.regular-expressions.info/lookaround.html

0 Karma

Explorer

Regex golf amrit? 🙂

0 Karma

SplunkTrust
SplunkTrust

Try to have this setting on Indexer and try.

0 Karma

Explorer

It wouldn't work on the indexer either. Since the data is structured, it's not parsed on an indexer, it goes straight into the index. Without parsing, there's no way to filter on events.

0 Karma