How to store a large search result set into a lookup table?

[ddrillic]

I have a large results set of a search which I would like to store as a lookup table. How can I do that?

[javiergn · Accepted solution]

Use the outputlookup command.
If large means millions of lines, you might be better off specifying .csv.gz in order to create a compressed file that you can later on read with inputlookup.

[ddrillic]

Great.

We ran:

index=provider tin!=" *" tin!="000000000"  tin="*" | dedup tin | fields - _raw | table * | outputlookup provider_lookup.csv

It created a 2.2 GBs file on the SH's file system and I can see it under the 'Lookup table files' as -
/opt/splunk/splunk/etc/apps/search/lookups/provider_lookup.csv

What should I do now?

[javiergn]

You can use it with either inputlookup or just lookup.
Keep in mind you will need to create a lookup in order to do that.
Take a look at the very detailed documentation about this topic.

Hope that helps.

[ddrillic]

Thank you javiergn - worked nicely but slow with a lookup table of 2 GBs and physical memory of 16 MBs.

The lookup part of the command looks like -
| lookup provider_lookup tin as prov_tin OUTPUT adr_ln_1_txt

[rusty009]

also be careful about saving large lookup files on your search head. You may bump into issues with syncing your bundle across indexers if you hit your maximum bundle size. You may need to whitelist the lookup folder within that app > https://answers.splunk.com/answers/3436/how-could-i-optimize-distributed-replication-of-large-lookup...

[ddrillic]

Thanks a lot!