topic Re: Inaccurate results from a lookup in Splunk Search

Inaccurate results from a lookup

jambajuice — Sat, 29 Jan 2011 10:43:01 GMT

I have a lookup table that contains CVSS vulnerability metrics. The fields are as follows:

"_time","cve_id",score,"access_vector","access_complexity",authentication,"integrity_impact","availability_impact","confidentiality_impact","vuln_product"

The vuln_product field is multivalued. Sometimes events may have 100+ items in the vuln_product field. Its data looks like the following:

"videolan:vlc_media_player:0.2.82
videolan:vlc_media_player:0.2.83
videolan:vlc_media_player:0.2.80"

The lookup table is about 11 MB in size. When I perform a search with about 50,000 results and I do a lookup on the cve_id and output the rest of those fields, data from the "vuln_product" field is showing up in other fields. I've double-checked the lookup table and the data looks clean. I've compared the entries in the lookup table with events that were or were not displaying properly and I can't see any difference in the data.

What might be the cause of Splunk not successfully getting events out of the lookup table?

Thx.

Craig

Re: Inaccurate results from a lookup

jambajuice — Tue, 01 Feb 2011 04:00:25 GMT

I tried splitting the lookup table into two 18,000 row (approximately) tables. When I performed the lookup on either table, the results were fine. If I combined the tables into a single one, data from the multi-valued field were still showing up in other fields.

Re: Inaccurate results from a lookup

jambajuice — Mon, 28 Sep 2020 09:23:45 GMT

I also tried creating two separate lookup tables. One contains the all of the fields except for the big, multi-valued "vuln_product" field. I created another lookup table that only has cve_id and vuln_product fields. If I do a lookup on the first table and then a lookup on the second table, everything is fine. But if I try and make a lookup on a single, big table, the data gets mashed up.

Is this a bug?

Re: Inaccurate results from a lookup

woodcock — Wed, 27 May 2015 13:20:46 GMT

My suspicion is that your lookup table has multiple key value entries. Assuming the first field is the only key field, check for key uniqueness like this:

awk -F, '{print $1}' ${SPLUNK_HOME}/etc/MyApp/lookups/MyLookupFile.csv | uniq -d