Apply multiple regular expressions from a lookup file

[lids4dt]

If I have a lookup containing a list of different regular expressions in a column, is there a way I can input the lookup and apply each regular expression to a search? (So as to avoid having to feed them in one by one.)

Something like this:

(sourcetype=sourcetype123) OR (sourcetype=sourcetype456) regex _raw= "LOOKUP_TABLE [inputlookup REGULAR_EXPRESSIONS.csv ]"

Thanks

[andygerber]

I got this to work, may not be the best way to do it, but it worked for me.
I just made a lookup with a single column, "name", containing the strings I'm searching for.

| inputlookup mylookup.csv
| map search="<your search>| eval search_str= \".*\" . $name$ . \".*\" " maxsearches=100000
| where match(<field from your search you're trying to match>, search_str)

Since I'm using this in the context of Enterprise Security and a notable search, the underlying search is only running over a small timeframe, and the lookup has only 20 elements. So it works pretty well. YMMV.

[malvidin]

Depending on the search, the number of results, and size of the lookup table, you can use map, inputlookup, and regex or eval/match.

<query> | table results | map [|inputlookup REGULAR_EXPRESSIONS.csv | regex REGEX=$results$ ] maxsearches=10

I'm going on my memory, do this might not be exactly right. If the lookup is smaller than the search, you can put the lookup first, and pipe that to map with your sourcetypes.

[lguinn2 · Accepted solution]

I don't see how that can work. However, lookups do support a wildcard (*) just like the search command. So you could do it like this:

(sourcetype=sourcetype123) OR (sourcetype=sourcetype456) 
| lookup my_lookup _raw as matchString OUTPUT match
| where match=1

This search keeps only the events that match at least one pattern in the csv file. It works if you do the following in your setup:

wildcard_lookup.csv

matchString,match
ab*c,1
ab*dX*,1

transforms.conf

[my_lookup]
filename = wildcard_lookup.csv
match_type = WILDCARD(matchString)
max_matches = 1
min_matches = 1
default_match = 0

Note that the value of the match field is always 1 in the csv file. But if there is no match, the value of the match field will be set to 0 - the value of the default_match setting in transforms.conf