Applying multiple regex to different fields (match_type REGEX for lookups)?

[wiederkehrc]

Hi,
we have a Data Model based search that we filter based on a lookup (with match_type WILDCARD) that matches different fields

| tstats count, values(Processes.dest) as dest, dc(Processes.dest) as dest_dc, min(_time) as earliest, max(_time) as latest, values(Processes.user) as user, dc(Processes.user) as user_dc from datamodel=Endpoint.Processes by Processes.process_guid Processes.parent_process_guid Processes.parent_process Processes.parent_process_path Processes.process Processes.process_path Processes.process_hash Processes.user 
| rex field=Processes.process_hash "MD5=(?<process_md5>[A-Z0-9]+)" 
| `drop_dm_object_name(Processes)` 
| lookup sysmon_rules parent_process parent_process_path process process_path process_md5 OUTPUT description score

This works well and saves uf from having multiple searches in place, but it would be great if there was something like a match_type REGEX for lookups.  We could then combine several entries in the lookup to one single line.
For example those 4 lines:

score,description,parent_process_path,parent_process,process_path,process,process_md5
80,Office: Execution MSHTA,C:\Program Files (x86)\Microsoft Office\root\Office*,*,*\mshta.exe,*,*
80,Office: Execution PWSH,C:\Program Files (x86)\Microsoft Office\root\Office*,*,*\powershell.exe,*,*
80,Office: Execution WSCRIPT,C:\Program Files (x86)\Microsoft Office\root\Office*,*,*\wscript.exe,*,*
80,Office: Execution CMD,C:\Program Files (x86)\Microsoft Office\root\Office*,*,*\cmd.exe,*,*

could be combined to:

score,description,parent_process_path,parent_process,process_path,process,process_md5
80,Office: Execution susp child,(?i)C:\Program Files (x86)\Microsoft Office\root\Office.*,.*,(cmd.exe|wscript.exe|powershell.exe|mshta.exe),.*,.*

We want to keep the possibility to match against multiple fields. Is there a trick (using inputlookup, map, ...) to optimize this?We're at a point where the lookup is getting cluttered because of small variations of processes in the endpoint data model we would like to alert on.

Hints, tips & help are appreciated.

Chris

[yuanliu]

Have you considered data normalization?  Instead of one lookup, use orthogonal lookups.

sysmon_rules

score	description	parent_process_path	parent_process	path_match	process	process_md5
80	Office: Execution susp child 1	C:\Program Files (x86)\Microsoft Office\root\Office*	*	susp 1	*	*

susp_child

process_path	susp_child
*\mshta.exe	susp 1
*\powershell.exe	susp 1
*\wscript.exe	susp 1
*\cmd.exe	susp 1
*\someother.exe	susp 2

Then, use this search

| tstats count, values(Processes.dest) as dest, dc(Processes.dest) as dest_dc,
 min(_time) as earliest, max(_time) as latest, values(Processes.user) as user,
 dc(Processes.user) as user_dc from datamodel=Endpoint.Processes
 by Processes.process_guid Processes.parent_process_guid Processes.parent_process
 Processes.parent_process_path Processes.process Processes.process_path
 Processes.process_hash Processes.user 
| rex field=Processes.process_hash "MD5=(?<process_md5>[A-Z0-9]+)" 
| `drop_dm_object_name(Processes)`
| lookup susp_child process_path output susp_child
| lookup sysmon_rules parent_process parent_process_path process susp_child process_md5 OUTPUT description score

[wiederkehrc]

Thank you for your suggestion. This solution works from a technical point of view and reduces some redundant information in the lookup.  The consequence ist, that analysts who maintain and modify the lookups will need to manage multiple lookups which adds a layer of complexity to the search. 

[yuanliu]

I thought analysts were such data nerds they could normalize a database with hands tied in the back😉

But OK if they are not.  All you need to do is to design a language that they can work with, then, produce the actual lookup programmatically.

For example,  they can use "|" to represent logical "OR" like in many programming languages, and input the following rule:

description	parent_process	parent_process_path	process	process_md5	process_path	score
Office: Execution susp child (cmd)	*	C:\Program Files (x86)\Microsoft Office\root\Office*	*	*	cmd.exe|wscript.exe|powershell.exe|mshta.exe	80

Save this to analyst_rule.csv.  Then run the following:

| inputcsv analyst_rule.csv
| eval process_path = split(process_path, "|")
| mvexpand process_path
| eval description = description . " (" . replace(process_path, "\..+", "") . ")" ``` this is perhaps unnecessary ```
| outputlookup real_rule

The real_rule table will look like

description	parent_process	parent_process_path	process	process_md5	process_path	score
Office: Execution susp child (cmd)	*	C:\Program Files (x86)\Microsoft Office\root\Office*	*	*	cmd.exe	80
Office: Execution susp child (wscript)	*	C:\Program Files (x86)\Microsoft Office\root\Office*	*	*	wscript.exe	80
Office: Execution susp child (powershell)	*	C:\Program Files (x86)\Microsoft Office\root\Office*	*	*	powershell.exe	80
Office: Execution susp child (mshta)	*	C:\Program Files (x86)\Microsoft Office\root\Office*	*	*	mshta.exe	80

This is just one of possible solutions to improve usability.