Getting Data In

Filtering Remote Event Log Collection Event Codes - a simple answer

Explorer

This is not a question, since when I tried to get a simple answer for what I believe is a simple problem, I could not find one that worked.

So for those of you who use a Windows Heavy Forwarder which collects Remote Event Logs from other Windows systems - you will likely want to filter a bunch of stuff so that it does not get forwarded to the indexer system.

There are a lot of things out there about version 6 and blacklisting and also about where and what to update on config files to filter this stuff. For me, neither support nor the other items mentioned on this site and the blog site seemed to work, at all. So if you have the above scenario, this is what worked for me. This may not be the politically correct way to do it, but all I know is after trying everything else, this is the only thing that worked for me ...

On your Windows based Heavy Forwarder that is collecting the event logs from remote systems:

Create and/or edit the file: \program files\splunk\etc\system\local\props.conf and add:

[WMI:WinEventLog:Security]
REPORT-rename-windows = windows-rename-user, windows-rename-user2, windows-rename-user3, windows-eventid, windows-group
REPORT-extract-windows = windows-src_ip, windows-src_port, windows-process, windows-src_nt_domain-1, windows-src_nt_domain-2, windows-src_nt_domain-3, windows-dest_nt_domain-1, windows-dest_nt_domain-2
TRANSFORMS-sec = setnull

[WMI:WinEventLog:System]
REPORT-rename-windows = windows-rename-user, windows-rename-user2, windows-rename-user3, windows-eventid, windows-group
REPORT-extract-windows = windows-src_ip, windows-src_port, windows-process, windows-src_nt_domain-1, windows-src_nt_domain-2, windows-src_nt_domain-3, windows-dest_nt_domain-1, windows-dest_nt_domain-2
TRANSFORMS-sys = setnull

[WMI:WinEventLog:Application]
REPORT-rename-windows = windows-rename-user, windows-rename-user2, windows-rename-user3, windows-eventid, windows-group
REPORT-extract-windows = windows-src_ip, windows-src_port, windows-process, windows-src_nt_domain-1, windows-src_nt_domain-2, windows-src_nt_domain-3, windows-dest_nt_domain-1, windows-dest_nt_domain-2
TRANSFORMS-app = setnull

The setnull line is the key (fairly standard splunk approach) and is referenced is the next file that you need:

\program files\splunk\etc\system\local\transforms.conf

[setnull]
REGEX=(?msi)(^EventType=3|^EventCode=(26|64|65|100|101|333|528|538|540|551|552|562|567|576|577|578|680|735|736|4114|2080|2334|1704|7035|7036|14500|10009))
DEST_KEY = queue
FORMAT = nullQueue

The EventCode above are what I filter and I have taken to be more aggressive and decide to filter (for now) anything that is of EventType=3 since that is a "success" and/or benign types of error messages, at least based on my thorough review of all the messages of this type that I have seen. I am not interested in things that work, I am interested in things that don't.

I wished that the blacklist feature in Splunk 6 would have worked. There was also some pointers to using wmi.conf which did not work and also some discussion about where to put the transforms.conf and props.conf files (apps\windows\local vs. etc\system\local) and the windows folder definitely did not work for me. Also, putting any of this on the indexer definitely did not work AT ALL.

Hope this is helpful to someone. Maybe one day splunk will do the common sense thing and write a simple explanation for this common need in this somewhat common scenario.

Path Finder

I have found SecureWork's document very useful for security event best practice.

`
[WinEventLog://Security]
disabled = 0
start_from = oldest
current_only = 0
evt_resolve_ad_obj = 1
checkpointInterval = 5
index = winevents

#based on SecureWorks Windows Log Monitoring Best Practices for Security and Compliance

whitelist = 517,520,529,530,531,532,533,534,535,536,537,539,540,560,565,566, 601,608,609,610,611,612,617,620,621,622,624,626,627, 628,629,630,631,632,633,634,635,636,637,638,639,641,642,643,644,645,646,647,648,649,650,651,652,653, 654,655,656,657,658,659,660,661,662,663,664,665,666,667,668,671,672,673,675,676,680,681,685, 1102,4616, 4624,4625,4656,4661,4662,4688,4697,4704,4705,4706,4707,4708,4713,4716,4717,4718,4719,4720, 4722,4723,4724,4725,4726,4727,4728,4729,4730,4731,4732,4733,4734,4735,4737,4738,4739,4740, 4741,4742,4743,4744,4745,4746,4747,4748,4749,4750,4751,4752,4753,4754,4755,4756,4757,4758, 4759,4760,4761,4762, 4763,4764,4767,4768,4769,4771,4776,4781,4906
`

Splunk Employee
Splunk Employee

The double edged sword is that Splunk can indeed read, parse and index just about anything... which means that the inherent flexibility that comes with that gift allows for multiple solutions to the same scenario. Confusing. Yes.

This is also why many answers come to you in philosophical form rather than in detail. Especially when the question uses mixed metaphors.

You have accomplished what you want to do, in the only way possible.

To filter specific events, traditionally you throw out what you don't want first (often the nullQueue stanza just says REGEX= .) Then you tell Splunk what you DO want to parse... in other words, throw out everything EXCEPT the following: and then you do another stanza but send that stuff to the parsingQueue.

What you've done, by specifying what you want to throw out and by default, keeping the rest is just another way to get it done... totally legit. It all just depends on what it is you're trying to accomplish. In your case, your list of "keepers" was quite long, so listing them made perfect sense. If you only wanted to keep two or three... then throwing away everything and then "keeping back" a short list would have been less typing for that job.

Apologies for the doc reference, but you can read about it here It's a high level overview but it tells you what order Splunk processes the data (there are more detailed explanations of this process... way more detailed. Links are available from this document) But this gives you the basics of why this works the way it does.

BLACKLIST

This is, I think why you got some "wishy washy" sounding responses here and there. Blacklist has nothing to do with what you were trying to do, so it just confused the subject.


whitelist/blacklist is never (that I can find or recall) used to filter within events. it is used in inputs.conf to filter filenames you don't or do want Splunk to monitor. It is used in outputs.conf to tell it to whitelist/blacklist specific indexes it is used in serverclass.conf as a filterType for servers/clients/machineTypes etc... in other words, a level ABOVE events...

So there isn't going to be a way to do what you want to do exclusively using blacklist no matter how many people you ask.

Bravo for finding the way to get it done on your own. What's interesting, is that you seem to be missing the fact that your example is almost exactly like the one in the doc! 🙂

Make no mistake. Your pain is felt, and the complexity of the learning curve is understood. We're working on ways to help educate folks, and simplify the process as much as possible. It does take time and practice. And it looks like you're doing great. Keep going!

With Splunk... the answer is always "YES!". It just might require more regex than you're prepared for!

Explorer

I did open a ticket and you responded to it with the typical, two different ways to do something and a link to more documentation which was generic. I included how I was trying to use blacklisting, amongst many things, and was given a pointer to the inputs.conf page - not terribly helpful.

I challenge anyone to give a simple configuration example for a heavy forwarder doing Remote Event Log collection and it works using only the blacklist feature. I could not get it to work based on what limited information was available for this scenario.

0 Karma

Communicator

Blacklist by eventcode is a feature of the WinEventLog input method, which you are not using. There is no way to make that work with WMI collection. The method specified in this post is the right way to do it for WMI.

0 Karma

Explorer

I appreciate the responses. For me, this was not "learning" how this works with filtering. I learned that there are eight different answers for the same question and many of which ended with "go to this article about filtering" which was a generic recipe. Ask 8 splunk experts, get eight different answers, some of which might work.

Splunk needs a better forum/method/cookbook approach to answering in a simple way how to do simple/common things. I have the cookbook book, which is great, but there are still other common things that are missing.

For example - how Remote Event Log collection works and is filtered is almost non-existent and many things talk about wmi.conf which has little to no documentation for its practical use.

The whole "blacklist" feature simply does not work as advertised, although it probably does in some esoteric way.

I hope my simple example helps some poor soul who is trying to get their stuff to work.

0 Karma

Splunk Employee
Splunk Employee

Hi if the blacklist/whitelist does not work, or is not well documented, do not hesitate to open a case and report it.

And I agree, WMI collection is different from WinEventLog collection in their mechanism, so they do not support the same methods.

0 Karma

Splunk Employee
Splunk Employee

Additional method to filter since Splunk 6.* at the forwarder level.
http://answers.splunk.com/answers/116817/filter-wineventlog-events-based-on-the-eventcodes

Splunk Employee
Splunk Employee

In addition to what Lisa has pointed out, you've also discovered that whitelist and blacklist filtering only applies to the forwardedindex filter at the filename level - which for your purposes is too high level. That technique is described in the same doc page (link provided in Lisa Guinn's comment above) in the section below the one describing the WMI nullqueue technique you've outlined in detail.

With Splunk... the answer is always "YES!". It just might require more regex than you're prepared for!

Legend

Some of the key things that you have discovered:

Parsing happens on a heavy forwarder. However, if you don't use a heavy forwarder (and most are Universal Forwarders), parsing happens on the indexer.

Filtering happens at parsing time, so your props.conf and transforms.conf must be located wherever the parsing occurs.

Documentation:

http://docs.splunk.com/Documentation/Splunk/6.0.1/Forwarding/Routeandfilterdatad#Filter_event_data_a...

http://wiki.splunk.com/Where_do_I_configure_my_Splunk_settings%3F

State of Splunk Careers

Access the Splunk Careers Report to see real data that shows how Splunk mastery increases your value and job satisfaction.

Find out what your skills are worth!