Getting Data In

Filtering Remote Event Log Collection Event Codes - a simple answer

redventures
Explorer

This is not a question, since when I tried to get a simple answer for what I believe is a simple problem, I could not find one that worked.

So for those of you who use a Windows Heavy Forwarder which collects Remote Event Logs from other Windows systems - you will likely want to filter a bunch of stuff so that it does not get forwarded to the indexer system.

There are a lot of things out there about version 6 and blacklisting and also about where and what to update on config files to filter this stuff. For me, neither support nor the other items mentioned on this site and the blog site seemed to work, at all. So if you have the above scenario, this is what worked for me. This may not be the politically correct way to do it, but all I know is after trying everything else, this is the only thing that worked for me ...

On your Windows based Heavy Forwarder that is collecting the event logs from remote systems:

Create and/or edit the file: \program files\splunk\etc\system\local\props.conf and add:

[WMI:WinEventLog:Security]
REPORT-rename-windows = windows-rename-user, windows-rename-user2, windows-rename-user3, windows-eventid, windows-group
REPORT-extract-windows = windows-src_ip, windows-src_port, windows-process, windows-src_nt_domain-1, windows-src_nt_domain-2, windows-src_nt_domain-3, windows-dest_nt_domain-1, windows-dest_nt_domain-2
TRANSFORMS-sec = setnull

[WMI:WinEventLog:System]
REPORT-rename-windows = windows-rename-user, windows-rename-user2, windows-rename-user3, windows-eventid, windows-group
REPORT-extract-windows = windows-src_ip, windows-src_port, windows-process, windows-src_nt_domain-1, windows-src_nt_domain-2, windows-src_nt_domain-3, windows-dest_nt_domain-1, windows-dest_nt_domain-2
TRANSFORMS-sys = setnull

[WMI:WinEventLog:Application]
REPORT-rename-windows = windows-rename-user, windows-rename-user2, windows-rename-user3, windows-eventid, windows-group
REPORT-extract-windows = windows-src_ip, windows-src_port, windows-process, windows-src_nt_domain-1, windows-src_nt_domain-2, windows-src_nt_domain-3, windows-dest_nt_domain-1, windows-dest_nt_domain-2
TRANSFORMS-app = setnull

The setnull line is the key (fairly standard splunk approach) and is referenced is the next file that you need:

\program files\splunk\etc\system\local\transforms.conf

[setnull]
REGEX=(?msi)(^EventType=3|^EventCode=(26|64|65|100|101|333|528|538|540|551|552|562|567|576|577|578|680|735|736|4114|2080|2334|1704|7035|7036|14500|10009))
DEST_KEY = queue
FORMAT = nullQueue

The EventCode above are what I filter and I have taken to be more aggressive and decide to filter (for now) anything that is of EventType=3 since that is a "success" and/or benign types of error messages, at least based on my thorough review of all the messages of this type that I have seen. I am not interested in things that work, I am interested in things that don't.

I wished that the blacklist feature in Splunk 6 would have worked. There was also some pointers to using wmi.conf which did not work and also some discussion about where to put the transforms.conf and props.conf files (apps\windows\local vs. etc\system\local) and the windows folder definitely did not work for me. Also, putting any of this on the indexer definitely did not work AT ALL.

Hope this is helpful to someone. Maybe one day splunk will do the common sense thing and write a simple explanation for this common need in this somewhat common scenario.

m4him7
Path Finder

I have found SecureWork's document very useful for security event best practice.

`
[WinEventLog://Security]
disabled = 0
start_from = oldest
current_only = 0
evt_resolve_ad_obj = 1
checkpointInterval = 5
index = winevents

#based on SecureWorks Windows Log Monitoring Best Practices for Security and Compliance

whitelist = 517,520,529,530,531,532,533,534,535,536,537,539,540,560,565,566, 601,608,609,610,611,612,617,620,621,622,624,626,627, 628,629,630,631,632,633,634,635,636,637,638,639,641,642,643,644,645,646,647,648,649,650,651,652,653, 654,655,656,657,658,659,660,661,662,663,664,665,666,667,668,671,672,673,675,676,680,681,685, 1102,4616, 4624,4625,4656,4661,4662,4688,4697,4704,4705,4706,4707,4708,4713,4716,4717,4718,4719,4720, 4722,4723,4724,4725,4726,4727,4728,4729,4730,4731,4732,4733,4734,4735,4737,4738,4739,4740, 4741,4742,4743,4744,4745,4746,4747,4748,4749,4750,4751,4752,4753,4754,4755,4756,4757,4758, 4759,4760,4761,4762, 4763,4764,4767,4768,4769,4771,4776,4781,4906
`

rsennett_splunk
Splunk Employee
Splunk Employee

The double edged sword is that Splunk can indeed read, parse and index just about anything... which means that the inherent flexibility that comes with that gift allows for multiple solutions to the same scenario. Confusing. Yes.

This is also why many answers come to you in philosophical form rather than in detail. Especially when the question uses mixed metaphors.

You have accomplished what you want to do, in the only way possible.

To filter specific events, traditionally you throw out what you don't want first (often the nullQueue stanza just says REGEX= .) Then you tell Splunk what you DO want to parse... in other words, throw out everything EXCEPT the following: and then you do another stanza but send that stuff to the parsingQueue.

What you've done, by specifying what you want to throw out and by default, keeping the rest is just another way to get it done... totally legit. It all just depends on what it is you're trying to accomplish. In your case, your list of "keepers" was quite long, so listing them made perfect sense. If you only wanted to keep two or three... then throwing away everything and then "keeping back" a short list would have been less typing for that job.

Apologies for the doc reference, but you can read about it here It's a high level overview but it tells you what order Splunk processes the data (there are more detailed explanations of this process... way more detailed. Links are available from this document) But this gives you the basics of why this works the way it does.

BLACKLIST

This is, I think why you got some "wishy washy" sounding responses here and there. Blacklist has nothing to do with what you were trying to do, so it just confused the subject.


whitelist/blacklist is never (that I can find or recall) used to filter within events. it is used in inputs.conf to filter filenames you don't or do want Splunk to monitor. It is used in outputs.conf to tell it to whitelist/blacklist specific indexes it is used in serverclass.conf as a filterType for servers/clients/machineTypes etc... in other words, a level ABOVE events...

So there isn't going to be a way to do what you want to do exclusively using blacklist no matter how many people you ask.

Bravo for finding the way to get it done on your own. What's interesting, is that you seem to be missing the fact that your example is almost exactly like the one in the doc! 🙂

Make no mistake. Your pain is felt, and the complexity of the learning curve is understood. We're working on ways to help educate folks, and simplify the process as much as possible. It does take time and practice. And it looks like you're doing great. Keep going!

With Splunk... the answer is always "YES!". It just might require more regex than you're prepared for!

redventures
Explorer

I did open a ticket and you responded to it with the typical, two different ways to do something and a link to more documentation which was generic. I included how I was trying to use blacklisting, amongst many things, and was given a pointer to the inputs.conf page - not terribly helpful.

I challenge anyone to give a simple configuration example for a heavy forwarder doing Remote Event Log collection and it works using only the blacklist feature. I could not get it to work based on what limited information was available for this scenario.

0 Karma

delink
Communicator

Blacklist by eventcode is a feature of the WinEventLog input method, which you are not using. There is no way to make that work with WMI collection. The method specified in this post is the right way to do it for WMI.

0 Karma

redventures
Explorer

I appreciate the responses. For me, this was not "learning" how this works with filtering. I learned that there are eight different answers for the same question and many of which ended with "go to this article about filtering" which was a generic recipe. Ask 8 splunk experts, get eight different answers, some of which might work.

Splunk needs a better forum/method/cookbook approach to answering in a simple way how to do simple/common things. I have the cookbook book, which is great, but there are still other common things that are missing.

For example - how Remote Event Log collection works and is filtered is almost non-existent and many things talk about wmi.conf which has little to no documentation for its practical use.

The whole "blacklist" feature simply does not work as advertised, although it probably does in some esoteric way.

I hope my simple example helps some poor soul who is trying to get their stuff to work.

0 Karma

yannK
Splunk Employee
Splunk Employee

Hi if the blacklist/whitelist does not work, or is not well documented, do not hesitate to open a case and report it.

And I agree, WMI collection is different from WinEventLog collection in their mechanism, so they do not support the same methods.

0 Karma

yannK
Splunk Employee
Splunk Employee

Additional method to filter since Splunk 6.* at the forwarder level.
http://answers.splunk.com/answers/116817/filter-wineventlog-events-based-on-the-eventcodes

rsennett_splunk
Splunk Employee
Splunk Employee

In addition to what Lisa has pointed out, you've also discovered that whitelist and blacklist filtering only applies to the forwardedindex filter at the filename level - which for your purposes is too high level. That technique is described in the same doc page (link provided in Lisa Guinn's comment above) in the section below the one describing the WMI nullqueue technique you've outlined in detail.

With Splunk... the answer is always "YES!". It just might require more regex than you're prepared for!

lguinn2
Legend

Some of the key things that you have discovered:

Parsing happens on a heavy forwarder. However, if you don't use a heavy forwarder (and most are Universal Forwarders), parsing happens on the indexer.

Filtering happens at parsing time, so your props.conf and transforms.conf must be located wherever the parsing occurs.

Documentation:

http://docs.splunk.com/Documentation/Splunk/6.0.1/Forwarding/Routeandfilterdatad#Filter_event_data_a...

http://wiki.splunk.com/Where_do_I_configure_my_Splunk_settings%3F

Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

Splunk is officially part of Cisco

Revolutionizing how our customers build resilience across their entire digital footprint.   Splunk ...

Splunk APM & RUM | Planned Maintenance March 26 - March 28, 2024

There will be planned maintenance for Splunk APM and RUM between March 26, 2024 and March 28, 2024 as ...