topic Bypassing lookup 1000 limit in Splunk Search

Bypassing lookup 1000 limit

cpm003 — Tue, 30 Jul 2019 08:26:13 GMT

Hi all,
I am trying to make a correlation between an inventory of assets and vulnerability indexed data.

I am currently using:

| lookup assets.csv vendor, product,version OUTPUT Hostname Delivery | where isnotnull (Delivery)

to get exactly which assets appear in the indexed data and are therefore vulnerable, however there is a limit of 1000 when using lookup files.

By trying to solve the problem, I have indexed the inventory data in "index = assets", how could I get the relation of vulnerable assets with data indexed in two different indexes?

Thanks in advance

Re: Bypassing lookup 1000 limit

jlemley — Tue, 30 Jul 2019 13:10:34 GMT

This is one of my favorite topics: Joining two data sets with the stats command.

A very quick and dirty way to do this with your data could be to try something like this:

index=vulnerability OR index=assets
| stats first(*) as * by vendor, product, version

This will pull your two data sources together, grouped (or joined) by the vendor, product, and version.

More information can be found here:
https://answers.splunk.com/answers/145077/how-to-perform-join-with-stats.html

There's also a great .conf18 presentation on this topic:
https://conf.splunk.com/watch/conf-online.html?search=join#/

Re: Bypassing lookup 1000 limit

cpm003 — Wed, 31 Jul 2019 13:40:02 GMT

Thank you very much for your response, although I have not be able to make it work, it wasn´t grouping data.

I have achieved it differently, I have created a new field in each index with:
| eval CPE = vendor + ":" + product + ":" version

then:
| stats values (index) as index values (Hostname) as Hostname by CPE | where mvcount (index) > 1 | mvexpand Hostname