Splunk Enterprise

Incomplete search results because of lookup definition's maxmatch limit in Splunk

mbasharat
Contributor

Hi,

I have a need for an alternative of | lookup abc field1 AS field2 OUTPUT field1, fieldA, fieldB, fieldC.

For above, I have a lookup definition from a lookup that holds information about more than 50,000 vulnerabilities. I am using this lookup definition in my queries and result set is no more than 1000. 1000 is the maxmatch limit of lookup definition that Splunk supports. I need an alternative e.g. a subsearch using lookup itself or anything that allows me to do match for all the values in my lookup which is approximately 50,000 on average as efficiently as possible.

Sample query (original query is much longer but I will  be using your provided solution to consolidate)

index=ABC sourcetype="XYZ"

`comment (This is to reduce Splunk's internal fields to keep my table size smaller)`
| fields - index, source, sourcetype, splunk_server, splunk_server_group, host, eventtype, field, linecount, punct, tag, tag::eventtype, _raw

`comment (This is to limit to the only fields which I need)`
| fields dns, vuln_id

`comment (vuln_id is a multivalued field and I have to separate them to get accurate stats. When stats is run, it takes care of expanding them and it works as expected)`
| makemv delim="," vuln_id

| stats count by vuln_id, dns

| lookup vuln_info VulnID AS vuln_id OUTPUT Scan_Type, OS, Environment


The below approach is what I have tried that is not returning anything but it should. I am missing something in this:

index=ABC sourcetype="XYZ"
| fields - index, source, sourcetype, splunk_server, splunk_server_group, host, eventtype, field, linecount, punct, tag, tag::eventtype, _raw
| fields dns, vuln_id
| makemv delim="," vuln_id
| stats count by vuln_id, dns
[| inputlookup vuln_info.csv
| fields VulnID, Scan_Type, OS, Environment
| rename VulnID as vuln_id]

Any solution that will work as efficiently as possible to get all records from lookup instead of incomplete dataset due to lookup definition's maxmatch limit of 1000 in Splunk. Thanks in-advance!!!

0 Karma

thambisetty
Super Champion
index=ABC sourcetype="XYZ" 
| fields dns, vuln_id 
| makemv delim="," vuln_id 
| stats count by vuln_id, dns 
| append 
    [| inputlookup vuln_info.csv 
    | fields VulnID, Scan_Type, OS, Environment 
    | rename VulnID as vuln_id]
| stats values(*) as * by vuln_id
————————————
If this helps, give a like below.
0 Karma

mbasharat
Contributor

Hi @ Thambisetty,

append has max results limit of 50,000. This also produces partial. Is there a join option?

0 Karma
Get Updates on the Splunk Community!

New Learning Videos on Topics Most Requested by You! Plus This Month’s New Splunk ...

Splunk Lantern is a customer success center that provides advice from Splunk experts on valuable data ...

How I Instrumented a Rust Application Without Knowing Rust

As a technical writer, I often have to edit or create code snippets for Splunk's distributions of ...

Splunk Community Platform Survey

Hey Splunk Community, Starting today, the community platform may prompt you to participate in a survey. The ...