Splunk Search

Attempt to workaround 10k subsearch limit -- how to combine multiple lookup files?

the_wolverine
Champion

I'm breaking up my search and outputting the results into separate files. How can I combine these files into a single file once I'm done? .. Using Splunk UI, of course 😉

Example would be something like: | inputlookup lookupfile1.csv lookupfile2.csv lookupfile3.csv | dedup fieldname

1 Solution

ziegfried
Influencer

The append flag of the inputlookup command to the rescue (inputcsv supports it as well)

| inputlookup lookupfile1.csv append=1 | inputlookup lookupfile2.csv append=1 | inputlookup lookupfile2.csv append=1 | dedup fieldname

View solution in original post

ziegfried
Influencer

The append flag of the inputlookup command to the rescue (inputcsv supports it as well)

| inputlookup lookupfile1.csv append=1 | inputlookup lookupfile2.csv append=1 | inputlookup lookupfile2.csv append=1 | dedup fieldname

Ayn
Legend

It would be interesting to hear more about why the 10000 limit is there in the first place. I've encountered it loads of time by now (using inputlookup for loading blacklists of IP's that should be matched for instance) and it's of course frustrating to be limited like this. The question is if there's a very good reason for this limit to exist and have this specific value, like for instance that past 10000 search terms you won't increase performance with any additional terms?

Get Updates on the Splunk Community!

Stay Connected: Your Guide to July and August Tech Talks, Office Hours, and Webinars!

Dive into our sizzling summer lineup for July and August Community Office Hours and Tech Talks. Scroll down to ...

Edge Processor Scaling, Energy & Manufacturing Use Cases, and More New Articles on ...

Splunk Lantern is a Splunk customer success center that provides advice from Splunk experts on valuable data ...

Get More Out of Your Security Practice With a SIEM

Get More Out of Your Security Practice With a SIEMWednesday, July 31, 2024  |  11AM PT / 2PM ETREGISTER ...