topic Re: Perform lookup on static file in Splunk Search

Perform lookup on static file

maxdessureault — Mon, 28 Sep 2020 09:45:20 GMT

I am using the following to extract two fields at search time, extract_domain and extract_ip

source="dns2.log" 
| rex "((?<extract_domain>(\w+(\(\d\))){1,}?)$)" 
| rex mode=sed field=extract_domain "s/(\\(\\d\\))/./g" 
| rex "(?<extract_ip>\b(?:(?:25[0-5]|2[0-4][0-9]|1[0-9][0-9]|[1-9]?[0-9])\.){3}(?:25[0-5]|2[0-4][0-9]|1[0-9][0-9]|[1-9]?[0-9])\b)" 
| rex mode=sed field=extract_domain "s/.$//"

I would like to lookup the extract_domain field against a .csv file which looks something like this

domain, status
splunk.com, good
facebook.com,bad
google.com, good
hi5.com, bad

I want to report on all domains with a status of bad.

I've followed the example for using lookup tables but it isnt working out.

Please help

Re: Perform lookup on static file

Ayn — Mon, 25 Jul 2011 06:19:46 GMT

What isn't working out? Lookup tables would indeed be ideal to use here.

Put your csv file in a "lookups" directory, for instance $SPLUNK_HOME/etc/system/lookups. On the first line, put the field names you want to use, for instance "domain,status". Call the file something, say, "domainstatus.csv".

Then refer to it in transforms.conf:

[domainstatus]
filename = domainstatus.csv

Finally refer to the lookup transform in props.conf:

[source::dns2.log]
LOOKUP-domainstatus = domainstatus domain OUTPUT status

After restarting Splunk you should now see a field "status" for each "domain" for the source "dns2.log".

Re: Perform lookup on static file

Lowell — Tue, 02 Aug 2011 14:11:59 GMT

I suspect the issue is the order in which splunk is performing the lookup vs field extractions. Since you are extracting fields in your search explicitly, then you will want to use the lookup command explicitly after your fields have been extracted via the rex commands. Otherwise, I believe that splunk attempt to do automatic lookups immediately after your base search (which is simply source="dns2.log", at which point in, extract_domain does not yet exist.)

I would suggest you try:

<your existing search> | lookup domainstatus domain OUTPUT status

Once you get this working (assuming you want this to be done for other searches too) then I would suggest moving your field extractions to configuration files and then enable an automatic lookup for your source (or more preferably, for your specific sourcetype.)

I'm struggling to find the exact search time operation order in the docs, or I'd link to it here.