How many inputlookup files can be read in a splunk query

[hanikawadhwa]

Hi Splunkers,

I have been given a requirement where I need to read more than 10k input lookup files to get some results. My question: is it advisable or feasible to read this number of input lookup files?
scenario: every lookup files showing the operational hours of each trains for last 10 days.

(mon)9-10 (tues) 11-12 (wed )8-9 (thu)7-6 (fri) 1-2 (mon) 3-3:30 (tue )10-18 (wed)7-10 (thur) 9-18 (fri) 4-9

query requires to read all these input lookup files to consider operational hours of all the 10k trains.

Is there any other way to read all these files in a single query

TIA

[niketn]

@hanikawadhwa seems a better use case for Single aggregated KV Store if at any given day you need to have data for 10K trains for 7 days i.e. 70K results.

Which implies inputlookup fileXXXXX.csv | outputlookup consolidatekvstore kind of scheduled search to read from lookup file and populate consolidated KVStore. There has to be another search to remove entry older than 7 days.

If this is a CSV file, other option would be to simple read the CSV file from the folder location where they are being written and upload to an index so that all this can be taken care automatically. With indexed_extraction, the tstats based query on index would perform way better on 70K results and even more as you can retain older data as per your disk utilization or retention requirement (even 1 year or more).

[hanikawadhwa]

Thank you so much. I understand that uploading to an index is a better way.