I have a large resultset, lookupb.csv which consists of about 4 million lines, that I'm searching against that I need to break up in order to bypass the 10k result limit in subsearch.
Example:
| inputlookup lookupa.csv | search NOT [ |inputlookup lookupb.csv | head 10000 | fields fieldname ]
Obviously, I have a way to get 10000 first lines, and maybe 10000 last lines, but what about in-between?
One option could be to have serial no in the lookup file itself or add serial no in subsearch and search based on it.
|inputlookup lookupb.csv | eval sno=1 | accum sno | where sno >= yourlowerlimit AND sno <= yourupperlimit
Also, consider option to increase the limit value in limits.conf so that there are less no of breakups.
That worked! Make it an answer so I can accept it.
One option could be to have serial no in the lookup file itself or add serial no in subsearch and search based on it.
|inputlookup lookupb.csv | eval sno=1 | accum sno | where sno >= yourlowerlimit AND sno <= yourupperlimit
Also, consider option to increase the limit value in limits.conf so that there are less no of breakups.