How do you match on multiple fields in a lookup table?

[tljohnson]

Hi all,

I've been banging my head against the wall trying to get this to work.

What I'm trying to do is to use a lookup table as a whitelist for detected security events. This worked great so far as long as I've only been matching on a single field, but I'd like to create more complex rules and it's falling flat.

Here's an example of what I've been using, which works great:

index=secevents
| lookup Threat_Feed_Whitelist md5 OUTPUTNEW iswhitelist as whitelist
| search NOT whitelist IN ("TRUE")
| table src_user cmdline md5 _time
| collect index=alerts source="Threat Feed"

So I analyze all incoming secevents, unless the md5 is in the whitelist, in which case we ignore it. All other events are sent to an alerting index. Right now Threat_Feed_Whitelist consists of columns md5, comments, and iswhitelist.

I'm trying to expand the whitelist to user, cmdline, md5, comments, and iswhitelist. A field might look like:

bob, *powershell*, *, "Patch management engineer", TRUE

If an event came in where the user was bob and the cmdline included "powershell", it should be whitelisted.

I'm not sure how to get this to match on all three fields though. Right now, it will whitelist anything (presumably because one of the columns is a wildcard). This is what I've been trying:

index=secevents
| lookup Threat_Feed_Whitelist user as src_user OUTPUTNEW iswhitelist as whitelist
| lookup Threat_Feed_Whitelist cmdline OUTPUTNEW iswhitelist as whitelist2
| lookup Threat_Feed_Whitelist md5 OUTPUTNEW iswhitelist as whitelist3
| search NOT ( whitelist IN ("TRUE") AND whitelist2 IN ("TRUE") AND whitelist3 IN ("TRUE") )
| table src_user cmdline md5 _time
| collect index=alerts source="Threat Feed"

Is there a way to do a lookup using multiple fields as the key?

[woodcock]

We did a similar thing and it is easier than you think. Define this macro:

[SIEMMacro_Filter_Analyst_Exclusions(2)]
args = typeNameString_INPUT, fieldPrefixString_INPUT
definition = [| inputlookup SIEMLookupAnalystFilter\
 | search Status = "Enabled"\
      AND   type ="$typeNameString_INPUT$"\
\
 | fields - TTL *Time* *time* Status count comment* Comment* Description Vendor Product ExclusionReason type _*\
 | rename * AS $fieldPrefixString_INPUT$*\
 | format\
 | rename COMMENT AS "Because this is always an exclusion, add a 'NOT' to the beginning"\
 | eval search = "NOT " . search]
errormsg = Description: Author=Gregg Woodcock
iseval = 0

Then define a lookup called SIEMLookupAnalystFilter.csv that has a type field for your type of filter, such as Authentication_Exclusions with the rows and columns that you need. Each row will create an OR and each column will create an AND. You use it like this:

You Search Here ...
AND `SIEMMacro_Filter_Analyst_Exclusions(Authentication_Exclusions, Authentication.)`
| Your
| Other
| Stuff
| Here

If you look at some of the other fields referenced in the macro, you will get an idea of other features that we built-in, too.

[tljohnson]

The answer was staring me in the face, but I had something else wrong so I couldn't see it. Using lookup with multiple fields works correctly, although I didn't think so at first.

index=secevents
| lookup Threat_Feed_Whitelist user as src_user cmdline md5 OUTPUTNEW iswhitelist as whitelist
| search NOT whitelist IN ("TRUE")
| table src_user cmdline md5 _time
| collect index=alerts source="Threat Feed"

I have come across a different problem relating to fields with asterisks in them not being matched correctly, but that's another problem.