Getting Data In

Do WinEventLog blacklists use AND or OR for a list of eventcodes?

a212830
Champion

Hi,

I am reading an Active Directory eventfeed, and it has an extensive blacklist (see below). Are these blacklists considered an "AND" or an "OR" ?

[WinEventLog://Security]
disabled = 0
start_from = oldest
evt_resolve_ad_obj = 0
checkpointInterval = 5
current_only = 1
blacklist = 4656-4658,4670,4776,4689-4695,4985,5058,5061,5145,5149-6999
blacklist1= EventCode="^4634|4661|4672|4688|476([8,9])|477([0,1,2])|5140$" Message="Account\sName:\s+(srvHPOM|MSGKVS
SRVAUXACCT|srvMSMQ_Admin|srvEVOrch|srvlocalrunner|srvroomview|srvgoodmobile|ANONYMOUS\sLOGON|MCLADCSRVACCT|CDMS|IUSR
_ECM_BACKOFFICE|(\w+\$))"
blacklist2= EventCode="^462([4,5])$" Message="Account\sName:\s+-*(srvHPOM|MSGKVSSRVAUXACCT|srvMSMQ_Admin|srvEVOrch|s
rvlocalrunner|srvroomview|srvgoodmobile|ANONYMOUS\sLOGON|MCLADCSRVACCT|CDMS|IUSR_ECM_BACKOFFICE|(\w+\$))"
index = ad
0 Karma

wcolgate_splunk
Splunk Employee
Splunk Employee

First, let us point out that in your example, you have both a "legacy" blacklist (only event code or code ranges), and two regex blacklists.

To interpret the processing of blacklists is as follows:

A blacklist key gets evaluated to produce a 1 or 0.

In the case of the legacy blacklist line, an event code is compared to the event code or event code ranges. If it matches an event code or falls into one of the ranges of event code, then blacklist line is computed as 1, otherwise 0.

In the case of the regex blacklist line, all terms are AND'ed together to compute the value of the blacklist line; if all terms match all regexes, then the blacklist line is computed as 1. If ANY regex fails, the entire blacklist line is computed as 0.

Splunk will process each blacklist line in order. If any of the blacklist lines is computed as 1, we stop there and toss out the event. As a programmer, one can consider them OR'd -- and with the optimization that once we find ONE blacklist line that computes 1, Splunk doesn't need to process any more blacklists, because once blacklisted, the event is gone.

Pro tip: make the blacklists that remove the largest amount of events have lower numbers (e.g. blacklist1, blacklist2 ...)

sloshburch
Splunk Employee
Splunk Employee

Great tip! It's parallel to writing a splunk search: reduce the events as easily as possible.

0 Karma

sloshburch
Splunk Employee
Splunk Employee

I think I get the confusion so I'll try to point out why it's hard to conceptualize.

References

The fields are explained here:
http://docs.splunk.com/Documentation/Splunk/6.4.2/Data/MonitorWindowseventlogdata#Disable_an_event_l...

But your specific question is demonstrated here: http://docs.splunk.com/Documentation/Splunk/6.4.2/Data/MonitorWindowseventlogdata#Create_advanced_fi...

At the end of that section, there is a bit that explains:


Note: You cannot specify an entry that has more than one key/regular expression set that references the same key. If, for example, you specify:

whitelist = EventCode="^1([0-5])$" EventCode="^2([0-5])$"

Splunk Enterprise ignores the first set and only attempts to include events that match the second set. In this case, only events that contain an EventCode between 20 and 25 match. Events that contain an EventCode between 10 and 15 do not match. Only the last set in the entry ever matches. To resolve this problem, specify two separate entries in the stanza:

whitelist = EventCode="^1([0-5])$"
whitelist1 = EventCode="^2([0-5])$"

Incorrect Interpretation

So that means events in whitelist OR whitelist1 will be kept (since as @woodcock pointed out, an event can't have multiple EventCodes). So now if we flip this to blacklist, we're talking:

blacklist = EventCode="^1([0-5])$"
blacklist1 = EventCode="^2([0-5])$"

which is the logical equivalent of:

whitelist = EventCode!="^1([0-5])$"
whitelist1 = EventCode!="^2([0-5])$"

which, according to what we concluded from the docs ( whitelist OR whitelist1), means:

EventCode!="^1([0-5])$" OR EventCode!="^2([0-5])$"

Pulling out the negatives, we get:

NOT ( EventCode="^1([0-5])$" AND EventCode="^2([0-5])$" )

BUT, this seems odd because how could such a thing ever be true.

Correct Interpretation (I hope)

So, I think the catch here is that since we're using blacklists, the behavior isn't as linearly logical (alliteration score!) as our instincts just walked us through. In other words, when we have multiple blacklists, events that match are eliminated like a funnel: remove events from the first blacklist, then use the result of that against the next list and so on.

So, wrapping things up:
Imagine your events coming in and nothing being filtered because there is no whitelist. Then we start the blacklists and we take the result set and reduce it by blacklist. Then take that result set (like a pipe in the SPL) and pass it against blacklist1. Then that results set against blacklist2 and so on. Therefore, the answer is that they are all ORs but it might be confusing to think of it that way and easier to imaging it like SPL where each blacklist acts as a filter. Ultimately, the final result should be this:
alt text

I'm going to post this page to the form at the bottom of the corresponding docs page to get them to validate and possibly update the docs to make this more clear.

0 Karma

woodcock
Esteemed Legend

It is OR because otherwise ( AND ), a single event would have to have more than one EventCode, which is impossible.

0 Karma

somesoni2
Revered Legend

If you think of blacklist as "NOT EventCode=..." then these are "AND". Basically all the blacklist criteria will be applied and any event matching any of the blacklist criteria will not be ingested.

0 Karma
Get Updates on the Splunk Community!

Optimize Cloud Monitoring

  TECH TALKS Optimize Cloud Monitoring Tuesday, August 13, 2024  |  11:00AM–12:00PM PST   Register to ...

What's New in Splunk Cloud Platform 9.2.2403?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.2.2403! Analysts can ...

Stay Connected: Your Guide to July and August Tech Talks, Office Hours, and Webinars!

Dive into our sizzling summer lineup for July and August Community Office Hours and Tech Talks. Scroll down to ...