Splunk Enterprise Security: How to drop all events on Exchange servers except for remote OWA access?

[milesbrennan]

We run a few Exchange servers and we need to collect logs for our Splunk Enterprise Security Suite, however, there are many webapps running on an Exchange server, we want to trim the logs we collect to only those which are presenting OWA externally.

Sample to keep:

2016-01-04 22:41:01 xxx.xxx.xxx.xxx GET /owa/auth/logon.aspx url=https://webmail.company.org/owa/user@company.org

We want to RegEx on "\/owa" above and keep these events, however there are multiple other web-directories (below) which we want to drop, as there is too much content using our licenses.

Sample to drop:

/Microsoft-Server-ActiveSync/default.eas
/Microsoft-Server-ActiveSync/Proxy
/EWS/Exchange.asmx
/EWS/mrsproxy.svc
/Autodiscover/Autodiscover.xml
/EWS/Exchange.ASMX

We are using the following settings for our props and transforms on the receiving indexers, however we are still seeing lots of log events for both OWA and non-OWA web URLs.

props.conf

[source:://C:\inetpub\logs\LogFiles]
TRANSFORMS-filter = clean, save_owa

[source:://D:\IISLogs\Logging]
TRANSFORMS-filter = clean, save_owa

transforms.conf

[clean]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[save_owa]
REGEX = \/owa
DEST_KEY = queue
FORMAT = indexQueue

I'm just wondering if anyone can advise why all our logs events are still being indexed, when we only want to keep the OWA events?

Regards

[milesbrennan · Accepted solution]

Seems we may have misunderstood how the "indexQueue" key is documented in the "transforms.conf" man page, and needed to insert the name of the index receiving the data... i.e. "main".

We now have a working solution which drops all incoming "IIS" events from the Exchange server, except for OWA activity logs. It still displays historical indexing correctly and our license / index usage has decreased.

Only the OWA events are indexed, all others are dropped. This should work for other data types, so hopefully others find this useful.

Regards.

props.conf

[source:://C:\inetpub\logs\LogFiles]
sourcetype = iis
[iis]
TRANSFORMS-filter = clean, save_owa

[source:://D:\IISLogs\Logging]
sourcetype = iis
[iis]
TRANSFORMS-filter = clean, save_owa

transforms.conf

[clean]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[save_owa]
REGEX = (?i:\/owa\/)              <-- Added case-insensitive search (optional)
DEST_KEY = queue
FORMAT = main                     <-- Not "indexQueue"

[aljohnson_splun]

Please mark your question as answered (by your answer) - glad you figured it out !

[lguinn2]

@milesbrennan - I believe that indexQueue would be fine in the last line of the transforms.conf. I think that the real problem is something else.
Maybe it is that the source:: stanza specified a directory. For some reason (which I don't know), the transform did not like that.
OR, it may have been that filter is not a unique class name; technically, each line in props.conf that specifies a transform should be unique.

Although your solution may work, I think that the change to props.conf is really what fixed the problem.

References:
Keep specific events and discard the rest - syntax for transforms.conf showing the indexQueue
props.conf in the Admin manual - the dirty details of props.conf - I have read this document at least 50 times in the last 6 years, and I learn something new every time I read it again. This is where I confirmed that the class name (class) in TRANSFORMS-class = xyz should be unique.