topic Re: Automatic Lookup matching on multiple fields in Splunk Search

Automatic Lookup matching on multiple fields

chillsgrove — Tue, 29 Sep 2020 17:40:26 GMT

I'm attempting to create an automatic lookup that matches src_ip, dest_ip, and signature in returns a "reason" and "status" field. The events always contain src_ip, dest_ip, and signature. In the lookup table, we may have src_ip, dest_ip, and/or signature but not all three are required.

Example lookup "whitelist.csv"
dest_ip,src_ip,signature,reason,status
8.8.8.8/32,0.0.0.0/0,456789,False Positive,Y
0.0.0.0/0,128.72.21.3,987654,Deemed Safe,Y
128.72.56.48,128.72.110.0/24,,Permitted,Y

I've already performed match typing in transforms.conf (see below)
[whitelist]
filename = whitelist.csv
match_type = CIDR(src_ip)
match_type = CIDR(dest_ip)
match_type = WILDCARD(signature)

What I need to happen is for the automatic lookup to match the "reason" and "status" fields based on the combination of src_ip, dest_ip, and signature (note: I'm using the CIDR notation 0.0.0.0/0 to indicate all IPs when the source or destination doesn't matter accordingly). I thought this solution would work but the automatic lookup just isn't functioning.

Any assistance provided is greatly appreciated.

Re: Automatic Lookup matching on multiple fields

maciep — Sat, 27 Jan 2018 16:06:54 GMT

what do you mean it isn't functioning? can you share the automatic lookup definition in props.conf?

also, why not have "*" in the lookup for signature when it doesn't matter, as opposed to not being there at all? If it should match any sig for that reason/status, then * might make more sense - you already have it defined as a wildcard match.

Re: Automatic Lookup matching on multiple fields

acharlieh — Tue, 29 Sep 2020 17:48:50 GMT

The first problem is your transforms.conf is incorrect. Each key in a stanza is unique, and if you set a key multiple times in a stanza the last one wins. Assuming if you took your transforms and put it into the local search app, you can use btool and see this to be the case:

$ ./bin/splunk btool transforms list whitelist --debug | grep -v system/default
./etc/apps/search/local/transforms.conf [whitelist]
./etc/apps/search/local/transforms.conf filename = whitelist.csv
./etc/apps/search/local/transforms.conf match_type = WILDCARD(signature)

Therefore only the WILDCARD on signature is taking effect. Turning and looking at the spec file in the docs we see match_type is a comma and space separated list so we can correct your transforms stanza like so:

[whitelist]
filename = whitelist.csv
match_type = CIDR(src_ip), CIDR(dest_ip), WILDCARD(signature)

After a restart/reload, we have the first problem fixed... and now we need to talk about the data in your example lookup... If you're using CIDR matching, fields must be in CIDR notation... (src_ip on the Deemed Safe row, and dest_ip on the Permitted row are both missing a /32 making them not CIDR fields, and thus fail to match).

Regarding the Permitted line.... If the intention on the was to match all signature values, the field needs to be a * and not a blank value.

With both the data and your definitions corrected, you should now able to verify the lookup is working correctly against sample data using the lookup command. Using as clauses in the OUTPUT part you can even see which row matched each value:

<base search> | lookup whitelist src_ip dest_ip signature OUTPUT src_ip AS lookup_src_ip dest_ip AS lookup_dest_ip signature as lookup_signature

After this, then to make the lookup automatic, you'd have to follow the docs to build it... or make the correct props.conf settings and restart/reload.

Re: Automatic Lookup matching on multiple fields

chillsgrove — Tue, 06 Feb 2018 03:52:03 GMT

Thank you. This worked perfectly.