Splunk Enterprise

Incomplete search results because of lookup definition's maxmatch limit in Splunk

mbasharat
Builder

Hi,

I have a need for an alternative of | lookup abc field1 AS field2 OUTPUT field1, fieldA, fieldB, fieldC.

For above, I have a lookup definition from a lookup that holds information about more than 50,000 vulnerabilities. I am using this lookup definition in my queries and result set is no more than 1000. 1000 is the maxmatch limit of lookup definition that Splunk supports. I need an alternative e.g. a subsearch using lookup itself or anything that allows me to do match for all the values in my lookup which is approximately 50,000 on average as efficiently as possible.

Sample query (original query is much longer but I will  be using your provided solution to consolidate)

index=ABC sourcetype="XYZ"

`comment (This is to reduce Splunk's internal fields to keep my table size smaller)`
| fields - index, source, sourcetype, splunk_server, splunk_server_group, host, eventtype, field, linecount, punct, tag, tag::eventtype, _raw

`comment (This is to limit to the only fields which I need)`
| fields dns, vuln_id

`comment (vuln_id is a multivalued field and I have to separate them to get accurate stats. When stats is run, it takes care of expanding them and it works as expected)`
| makemv delim="," vuln_id

| stats count by vuln_id, dns

| lookup vuln_info VulnID AS vuln_id OUTPUT Scan_Type, OS, Environment


The below approach is what I have tried that is not returning anything but it should. I am missing something in this:

index=ABC sourcetype="XYZ"
| fields - index, source, sourcetype, splunk_server, splunk_server_group, host, eventtype, field, linecount, punct, tag, tag::eventtype, _raw
| fields dns, vuln_id
| makemv delim="," vuln_id
| stats count by vuln_id, dns
[| inputlookup vuln_info.csv
| fields VulnID, Scan_Type, OS, Environment
| rename VulnID as vuln_id]

Any solution that will work as efficiently as possible to get all records from lookup instead of incomplete dataset due to lookup definition's maxmatch limit of 1000 in Splunk. Thanks in-advance!!!

0 Karma

thambisetty
SplunkTrust
SplunkTrust
index=ABC sourcetype="XYZ" 
| fields dns, vuln_id 
| makemv delim="," vuln_id 
| stats count by vuln_id, dns 
| append 
    [| inputlookup vuln_info.csv 
    | fields VulnID, Scan_Type, OS, Environment 
    | rename VulnID as vuln_id]
| stats values(*) as * by vuln_id
————————————
If this helps, give a like below.
0 Karma

mbasharat
Builder

Hi @ Thambisetty,

append has max results limit of 50,000. This also produces partial. Is there a join option?

0 Karma
Get Updates on the Splunk Community!

Introducing the Splunk Community Dashboard Challenge!

Welcome to Splunk Community Dashboard Challenge! This is your chance to showcase your skills in creating ...

Built-in Service Level Objectives Management to Bridge the Gap Between Service & ...

Wednesday, May 29, 2024  |  11AM PST / 2PM ESTRegister now and join us to learn more about how you can ...

Get Your Exclusive Splunk Certified Cybersecurity Defense Engineer Certification at ...

We’re excited to announce a new Splunk certification exam being released at .conf24! If you’re headed to Vegas ...