Splunk Search

Filtering on UF for Specific Events then Delete the Rest (6.3.2)

markschoonover
Explorer

Hello Splunkers,

I've been working on filtering IIS events. What I need to keep is any event that contains auth.owa, then nullQueue the rest. I've been through the docs many times but something is missing. I've been able to send all events to nullQueue no problem but I've not been able to let the wanted events through.

This is the message I'm interested in:

2016-03-07 22:39:02 127.0.0.1 POST /OWA/auth.owa &CorrelationID=<empty>;&ClientId=AODTP0KGF0BGDQLLSW&cafeReqId=a5e157b9-047b-4e2d-a486-809dc938ed16; 443 foobar@xxx.yyy.com 127.0.0.1 AMProbe/Local/ClientAccess 200 0 0 62

Here's my props.conf file:

[source::...\\IISLogs\\...\\*]
TRANSFORMS-iis = iisbitbucket,iisauthonly

Transforms looks like this:

[iisbitbucket]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[iisauthonly]
REGEX = auth\.owa
DEST_KEY = queue
FORMAT = indexQueue

My gut is telling me I'm missing something fundamental here but I can't figure out what. I've tried all kinds of combinations of REGEX in the iisauthonly stanza but so far, nothing works. The results of this config is all messages are making it into the indexQueue. These conf files are located on the UFs which are Windows 2008 R2 servers. According to http://docs.splunk.com/Documentation/Splunk/6.2.3/Forwarding/Routeandfilterdatad#Keep_specific_event...

scroll to the bottom, structured data is filtered on the UF. Any ideas on what I'm missing?

1 Solution

markschoonover
Explorer

With some help from amrit on IRC, here's the config that allows filtering IIS structured data on a UF:

props.conf

[source::...\\IISLogs\\...\\*]
TRANSFORMS-iis = iisbitbucket,iisauthonly

Order of transforms stanzas is important. iisbitbucket drops everything, then iisauthonly is used to filter on the string /owa/auth.owa

transforms.conf

[iisbitbucket]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[iisauthonly]
SOURCE_KEY = field:cs_uri_stem
REGEX = (?i)/owa/auth.owa
DEST_KEY = queue
FORMAT = parsingQueue

Set the SOURCE_KEY to the field you need to filter. The /owa/auth.owa is found (as well as many other stems) in the cs_uri_stem field of the IIS structured data. REGEX is what to match in the field, DEST_KEY has to be set to queue. Some instructions about filtering on an indexer will have a FORMAT=index which is what I followed at first. After reading on all the different queues in a UF, I needed to feed the matched results back into the parsingQueue so the UF can push that to the indexer.

View solution in original post

markschoonover
Explorer

With some help from amrit on IRC, here's the config that allows filtering IIS structured data on a UF:

props.conf

[source::...\\IISLogs\\...\\*]
TRANSFORMS-iis = iisbitbucket,iisauthonly

Order of transforms stanzas is important. iisbitbucket drops everything, then iisauthonly is used to filter on the string /owa/auth.owa

transforms.conf

[iisbitbucket]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[iisauthonly]
SOURCE_KEY = field:cs_uri_stem
REGEX = (?i)/owa/auth.owa
DEST_KEY = queue
FORMAT = parsingQueue

Set the SOURCE_KEY to the field you need to filter. The /owa/auth.owa is found (as well as many other stems) in the cs_uri_stem field of the IIS structured data. REGEX is what to match in the field, DEST_KEY has to be set to queue. Some instructions about filtering on an indexer will have a FORMAT=index which is what I followed at first. After reading on all the different queues in a UF, I needed to feed the matched results back into the parsingQueue so the UF can push that to the indexer.

amrit
Splunk Employee
Splunk Employee

You can probably reduce to one regex/transform by using a negative lookahead:

REGEX=(?!/owa/auth.owa)

Haven't verified it, but... it should work. See more at: http://www.regular-expressions.info/lookaround.html

0 Karma

markschoonover
Explorer

Regex golf amrit? 🙂

0 Karma

somesoni2
Revered Legend

Try to have this setting on Indexer and try.

0 Karma

markschoonover
Explorer

It wouldn't work on the indexer either. Since the data is structured, it's not parsed on an indexer, it goes straight into the index. Without parsing, there's no way to filter on events.

0 Karma
Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

ICYMI - Check out the latest releases of Splunk Edge Processor

Splunk is pleased to announce the latest enhancements to Splunk Edge Processor.  HEC Receiver authorization ...

Introducing the 2024 SplunkTrust!

Hello, Splunk Community! We are beyond thrilled to announce our newest group of SplunkTrust members!  The ...