I'm attempting to create an automatic lookup that matches srcip, destip, and signature in returns a "reason" and "status" field. The events always contain srcip, destip, and signature. In the lookup table, we may have srcip, destip, and/or signature but not all three are required.
Example lookup "whitelist.csv"
I've already performed match typing in transforms.conf (see below)
filename = whitelist.csv
matchtype = CIDR(srcip)
matchtype = CIDR(destip)
match_type = WILDCARD(signature)
What I need to happen is for the automatic lookup to match the "reason" and "status" fields based on the combination of srcip, destip, and signature (note: I'm using the CIDR notation 0.0.0.0/0 to indicate all IPs when the source or destination doesn't matter accordingly). I thought this solution would work but the automatic lookup just isn't functioning.
Any assistance provided is greatly appreciated.
what do you mean it isn't functioning? can you share the automatic lookup definition in props.conf?
also, why not have "*" in the lookup for signature when it doesn't matter, as opposed to not being there at all? If it should match any sig for that reason/status, then * might make more sense - you already have it defined as a wildcard match.
The first problem is your transforms.conf is incorrect. Each key in a stanza is unique, and if you set a key multiple times in a stanza the last one wins. Assuming if you took your transforms and put it into the local search app, you can use btool and see this to be the case:
$ ./bin/splunk btool transforms list whitelist --debug | grep -v system/default ./etc/apps/search/local/transforms.conf [whitelist] ./etc/apps/search/local/transforms.conf filename = whitelist.csv ./etc/apps/search/local/transforms.conf match_type = WILDCARD(signature)
Therefore only the WILDCARD on signature is taking effect. Turning and looking at the spec file in the docs we see match_type is a comma and space separated list so we can correct your transforms stanza like so:
[whitelist] filename = whitelist.csv match_type = CIDR(src_ip), CIDR(dest_ip), WILDCARD(signature)
After a restart/reload, we have the first problem fixed... and now we need to talk about the data in your example lookup... If you're using CIDR matching, fields must be in CIDR notation... (srcip on the Deemed Safe row, and destip on the Permitted row are both missing a
/32 making them not CIDR fields, and thus fail to match).
Regarding the Permitted line.... If the intention on the was to match all signature values, the field needs to be a
* and not a blank value.
With both the data and your definitions corrected, you should now able to verify the lookup is working correctly against sample data using the lookup command. Using as clauses in the OUTPUT part you can even see which row matched each value:
<base search> | lookup whitelist src_ip dest_ip signature OUTPUT src_ip AS lookup_src_ip dest_ip AS lookup_dest_ip signature as lookup_signature
After this, then to make the lookup automatic, you'd have to follow the docs to build it... or make the correct props.conf settings and restart/reload.