Need to eliminate over 90% of forwarded events before reaching indexer. What is the CPU load on a UF?

[rob_gibson]

I have an indexer license being overwhelmed by useless Windows UF event forwarding (across 40+ UFs). I have determined that upwards of 90% of these events are "noise", and need to drastically reduce the amount of events hitting the indexer, thereby reducing our high licensing and storage requirement. In some cases each forwarder produces over 300 events per second.

I do not have the ability to stop the events from being created, but I believe that using a blacklist of about 7-10 event codes I can stop about 95% of the events from being sent to the indexer.

Will a blacklist that blocks 95% of events being forwarded overload the forwarder CPU? I'm not clear on when the heavy forwarder must be used.

[ekost · Accepted solution]

There is excellent support for blacklisting Windows Event Log contents with a UF as of version 6+. References in answers and a blog. As to anticipated CPU load, in my experience dropping events via the UF won't use more CPU than the reading and parsing tasks already consume. I also recommend testing when implementing blacklists, as you may not be able to retrieve events that were "accidentally" dropped.

[rob_gibson]

Thanks for the links. I had reviewed them before posting. However, I did recall seeing something about a best practices suggestion to use a heavy forwarder beyond 50%, but I can't find that post. After much searching I can't find the answer I'm looking for.

Because I have so many forwarders sending junk to my indexer, I assumed that filtering on the UF's was the best approach.

The events I want to blacklist are not on anybody's standard "recommended" security events list. I won't miss them.

[ekost]

Very cool! There is a ton of old references to using a heavy forwarder as that was really the only way to accomplish this task prior to version 6.

The gist was: if you're dropping 50+% of the events, why do that on the indexers? You can install a HF on that Windows box and skip placing that data on the wire. Bonus, you won't clog your indexer with regexes that will review all of the incoming windows data.

In your use-case, with current forwarders.. use the cool new feature!

[rob_gibson]

I am in the process of upgrading my indexer to 6.2.0 for AIX but the readme advice was that UF's never need to be upgraded so it was not in my plans. I am assuming that I must upgrade my UF's to take advantage of the version 6 filtering features you are referring to.

Time to edit that RFC I think...

[ekost]

Yes. The BL / WL feature is version 6+. If you have Windows UF pre- 6.0, you now have a compelling reason to upgrade them.

[harsmarvania57]

On heavy forwarder you can use props.conf and transforms.conf to filter out events and then it will send to Indexers. But you can't filter events on Universal forwarder with props.conf and transforms.conf.