topic Re: Problem with filtering events prior to indexing in Getting Data In

Problem with filtering events prior to indexing

ferdydek — Mon, 28 Sep 2020 15:31:59 GMT

I need to drop "Filtering Platform Connections", I also want to drop most of "Audit File System" events from windows servers. I do not have control over source servers, those only have universal-forwarder. I can only manipulate the indexer-all-in-one Splunk server.

I followed http://docs.splunk.com/Documentation/Splunk/5.0.6/Deploy/Routeandfilterdatad and some posts on answers and was testing the following:

in ../etc/system/local/

props.conf

[WinEventLog:Security]
TRANSFORMS-wmi = NullEvents-null, WinSecEvents-null

transforms.conf

[NullEvents-null]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[WinSecEvents-null]
REGEX=(?msi)^EventCode=(5156|104)\D
DEST_KEY = queue
FORMAT = indexQueue

Unfortunately the filtered events still show up.
What I did so far:

tested the regex on search in splunk - works,
tried adding the files to /etc/apps/windows/ /etc/apps/Splunk_TA_Windows /splunk/app/Splunk_for_Exchange as those were the files that ./splunk cmd btool and grep came up with,
tried several source names [WMI:WinEventLog:Security] [source::WinEventLog:Security] [source::WMI:WinEventLog:Security] [source::main] [WMI:WinEventLog:Security]
rebooting after each test (ofc),
created a case with splunk and gave them the dump file, but this is not solved yet.

I'm wondering if I'm putting the files in proper place (folder), if I'm using the correct [source] ?

++ I'm on 5.0.6. ++ I'm fresh to Splunk configurations ++ I'm abusing my license as I cant filter the events, so the issue is urgent ++

Re: Problem with filtering events prior to indexing

lukejadamec — Sun, 22 Dec 2013 12:21:37 GMT

A few things to help troubleshooting.

1) Put the props and transforms in etc//system//local. This directory has the highest precedence for index time extractions.

2) Use the source as opposed to the sourcetype. Extractions for the source will take precedence over extractions for sourcetype, so if there is a conflict it will only be with source.

3) You should not guess at the source. Run a search on the data and verify the source from the source field, and use it exactly preceded by [source::<yoursourcehere>].

4) If you want to keep EventCode=(5156|104)\D, then why call them null?
5) The transform requires an indexer restart to take effect and will only affect newly indexed logs.

Update: config to drop two event codes and keep everything else.

Have you tried this?

Make sure the source is correct, then in ../etc/system/local/

props.conf

[WinEventLog:Security]
TRANSFORMS-wmi = WinSecEvents-null

transforms.conf

[WinSecEvents-null]
REGEX=(?msi)^EventCode=(5156|104)\D
DEST_KEY = queue
FORMAT = nullQueue

Re: Problem with filtering events prior to indexing

ferdydek — Sun, 22 Dec 2013 13:21:06 GMT

Thanks Luke!
Since I followed the docs and some other posts here and it didn't work for me, thus those try&error steps :]

This is the first location I was adding the files. Not resolved.
Yup, the source name comes from searches. In the particular example source is identical to sourcetype.
For some crazy reason cant edit the post (captcha failes every time). The 5156|104 is in reality going also to nullQueue . Yes, I want to filter them out, not in.
Correct, restarted, each time waiting for new logs to get indexed to see if worked.

Re: Problem with filtering events prior to indexing

lukejadamec — Mon, 23 Dec 2013 18:26:31 GMT

If that is the case, then your config is off the mark.

When I want to drop particular Windows events in Splunk 5, I use just one stanza. Try dropping all reference and stanzas to NullEvents-null, and keep only WinSecEvents-null.

My guess is the config you have posted would drop everything except those two events even though you named it -null.

Re: Problem with filtering events prior to indexing

ferdydek — Thu, 26 Dec 2013 11:32:07 GMT

I do want to drop those 2 events, thus they go to nullQueue. I'm pretty confident this is how it should be.

I also forgot to mention - splunk is running as a "splunk" user, and yes, all config files are owned by "splunk" user.

Re: Problem with filtering events prior to indexing

lukejadamec — Thu, 26 Dec 2013 12:51:34 GMT

The configs you posted won't do what you want. I know you said you could not edit them, but when you say "I pretty confident this is how it should be" it is kind of confusing - what do you mean by 'this'?
I updated my answer to remove the first nullQueue stanza and changed to indexQueue to nullQueue for the events you want to drop.