topic Re: Filter windows events on indexer from a universal forwarder in Getting Data In

Filter windows events on indexer from a universal forwarder

jstockamp — Thu, 12 May 2011 23:56:18 GMT

I've been beating my head against a wall with this for a few hours. My setup is I have a linux indexer with a few windows universal forwarders sending data to it. I'm trying to filter out some event codes from the index. I'm assuming that since these are universal forwarders, I'll need to forward all events and filter out the ones I don't want at index time, correct?

Based on that assumption on the indexer I've got the following in $SPLUNK_HOME/etc/system/local/props.conf

[WinEventLog:Security]

TRANSFORMS-Filter_Events = FilterSecurityEvents

and in $SPLUNK_HOME/etc/system/local/transforms.conf

[FilterSecurityEvents]

REGEX=(?msi)^EventCode=(4634|4624|4769)

DEST_KEY=queue

FORMAT=nullQueue

I'm still getting events with these 3 codes in the index. Any idea(s) why this isn't working?

Re: Filter windows events on indexer from a universal forwarder

Ron_Naken — Fri, 13 May 2011 17:42:58 GMT

If you remove your ^ anchor, it should work:

REGEX=EventCode=(4634|4624|4769)

EDIT: Don't forget to restart Splunk after making/changing an index-time transform.

Re: Filter windows events on indexer from a universal forwarder

Ron_Naken — Fri, 13 May 2011 17:44:53 GMT

You can test your RegEx in Search, before you hit the drawing board with props/transforms. Use: ... | regex "EventCode=(4634|4624|4769)" to make sure your events show.

Re: Filter windows events on indexer from a universal forwarder

jstockamp — Fri, 13 May 2011 18:44:50 GMT

This is still not working. Here's my props.conf from /opt/splunk/etc/system/local:

[host::web002.xxx]
TRANSFORMS-no_lb = no_lb_traffic_transform
[WinEventLog:Security]
TRANSFORMS-FilterEvents = filtersecurityevents

and here's the transforms.conf file from the same directory:

[no_lb_traffic_transform]
REGEX=(10\.100\.0\.3|10\.100\.0\.2)(.*) 
DEST_KEY=queue
FORMAT=nullQueue
[filtersecurityevents]
REGEX=EventCode=(4634|4624|4769)
DEST_KEY=queue
FORMAT=nullQueue

As you can see, I have an additional stanza in each from some filtering I'm doing on some other web logs. That filtering is working fine, it's just the windows events that aren't being filtered. I have tested my regex in the search app and it is matching all the events that I want to filter out.

Any help would be greatly appreciated.

Re: Filter windows events on indexer from a universal forwarder

Ron_Naken — Fri, 13 May 2011 19:49:54 GMT

I tried your stanzas in my lab, and the events are being filtered properly. I tested with your original RegEx (which is more efficient than using no anchor, but doesn't account for possible odd multi-line behavior). You might contact support to see if there is an issue when using a forwarder. One other option might be to try specifying the transform using host or source, to see if it's limited to sourcetype.

One other consideration: My lab is eating the security eventlog as a file, rather than using WMI. The structure is exactly the same; however, the issue may have something to do with the input.

Cheers

Re: Filter windows events on indexer from a universal forwarder

jstockamp — Fri, 13 May 2011 20:18:04 GMT

Thanks. I've opened a case with support. I've tried specifying a specific host and source in props.conf instead of using a sourcetype and it still didn't work.

Re: Filter windows events on indexer from a universal forwarder

jstockamp — Fri, 13 May 2011 20:55:57 GMT

I believe I've solved this. The issue is that the forwarder was running version 4.2.1 of the universal forwarder and my indexer was still running splunk version 4.2. I upgraded the indexer to 4.2 and it now seems to be working and filtering correctly.

Kind of odd that there were no errors in splunkd.log and that the forwarder clearly had no problems talking to the indexer since events were still making it into the index (just not being filtered).

Re: Filter windows events on indexer from a universal forwarder

gwege — Wed, 30 Oct 2013 17:47:27 GMT

Any update to this? I'm having a similar issue.