Incomplete search results because of lookup definition's maximum match limit in Splunk

[mbasharat]

Re-initiation of an older question I had asked:

Hi,

I have a need for an alternative of | lookup abc field1 AS field2 OUTPUT field1, fieldA, fieldB, fieldC.

For above, I have a lookup definition from a lookup that holds information about more than 50,000 vulnerabilities. I am using this lookup definition in my queries and result set is no more than 1000. 1000 is the maxmatch limit of lookup definition that Splunk supports. I need an alternative e.g. a subsearch using lookup itself or anything that allows me to do match for all the values in my lookup which is approximately 50,000 on average as efficiently as possible.

Sample query (original query is much longer but I will  be using your provided solution to consolidate)

index=ABC sourcetype="XYZ"

`comment (This is to reduce Splunk's internal fields to keep my table size smaller)`
| fields - index, source, sourcetype, splunk_server, splunk_server_group, host, eventtype, field, linecount, punct, tag, tag::eventtype, _raw

`comment (This is to limit to the only fields which I need)`
| fields dns, vuln_id

`comment (vuln_id is a multivalued field and I have to separate them to get accurate stats. When stats is run, it takes care of expanding them and it works as expected)`
| makemv delim="," vuln_id

| stats count by vuln_id, dns

| lookup vuln_info VulnID AS vuln_id OUTPUT Scan_Type, OS, Environment

The below approach is what I have tried that is not returning anything but it should. I am missing something in this:

index=ABC sourcetype="XYZ"
| fields - index, source, sourcetype, splunk_server, splunk_server_group, host, eventtype, field, linecount, punct, tag, tag::eventtype, _raw
| fields dns, vuln_id
| makemv delim="," vuln_id
| stats count by vuln_id, dns
[| inputlookup vuln_info.csv
| fields VulnID, Scan_Type, OS, Environment
| rename VulnID as vuln_id]

Any solution that will work as efficiently as possible to get all records from lookup instead of incomplete dataset due to lookup definition's maxmatch limit of 1000 in Splunk. Thanks in-advance!!!

[scelikok · Accepted solution]

Hi @mbasharat,

Can you try appendpipe ?

index=ABC sourcetype="XYZ"
| fields - index, source, sourcetype, splunk_server, splunk_server_group, host, eventtype, field, linecount, punct, tag, tag::eventtype, _raw
| fields dns, vuln_id
| makemv delim="," vuln_id
| stats count by vuln_id, dns
| appendpipe
[| inputlookup max=0 vuln_info.csv
| fields VulnID, Scan_Type, OS, Environment
| rename VulnID as vuln_id]
| stats count values(dns) as dns values(Scan_Type) as Scan_Type values(OS) as OS values(Environment) as Environment by vuln_id

[mbasharat]

@ scelikok,

Doing some testing so will respond back shortly.

[scelikok]

Hi @mbasharat,

The way you used inputlookup in your second search is not matching but filtering. Please try below;

index=ABC sourcetype="XYZ"
| fields - index, source, sourcetype, splunk_server, splunk_server_group, host, eventtype, field, linecount, punct, tag, tag::eventtype, _raw
| fields dns, vuln_id
| makemv delim="," vuln_id
| stats count by vuln_id, dns
| append
[| inputlookup vuln_info.csv
| fields VulnID, Scan_Type, OS, Environment
| rename VulnID as vuln_id]
| stats count values(dns) as dns values(Scan_Type) as Scan_Type values(OS) as OS values(Environment) as Environment by vuln_id

[mbasharat]

Hi @scelikok,

Append has max results limit of 50,000. I have a lot more than this. Any join option? Thanks.