https://community.splunk.com/t5/Splunk-Search/How-to-store-a-large-search-result-set-into-a-lookup-table/m-p/215973#M63342 <P>Use the <A href="http://docs.splunk.com/Documentation/Splunk/6.4.0/SearchReference/Outputlookup">outputlookup</A> command.<BR /> If large means millions of lines, you might be better off specifying .csv.gz in order to create a compressed file that you can later on read with inputlookup.</P> Mon, 18 Apr 2016 18:24:46 GMT javiergn 2016-04-18T18:24:46Z https://community.splunk.com/t5/Splunk-Search/How-to-store-a-large-search-result-set-into-a-lookup-table/m-p/215972#M63341 <P>I have a large results set of a search which I would like to store as a lookup table. How can I do that? </P> Mon, 18 Apr 2016 18:21:51 GMT https://community.splunk.com/t5/Splunk-Search/How-to-store-a-large-search-result-set-into-a-lookup-table/m-p/215972#M63341 ddrillic 2016-04-18T18:21:51Z https://community.splunk.com/t5/Splunk-Search/How-to-store-a-large-search-result-set-into-a-lookup-table/m-p/215973#M63342 <P>Use the <A href="http://docs.splunk.com/Documentation/Splunk/6.4.0/SearchReference/Outputlookup">outputlookup</A> command.<BR /> If large means millions of lines, you might be better off specifying .csv.gz in order to create a compressed file that you can later on read with inputlookup.</P> Mon, 18 Apr 2016 18:24:46 GMT https://community.splunk.com/t5/Splunk-Search/How-to-store-a-large-search-result-set-into-a-lookup-table/m-p/215973#M63342 javiergn 2016-04-18T18:24:46Z https://community.splunk.com/t5/Splunk-Search/How-to-store-a-large-search-result-set-into-a-lookup-table/m-p/215974#M63343 <P>Great. </P> <P>We ran:</P> <PRE><CODE>index=provider tin!=" *" tin!="000000000" tin="*" | dedup tin | fields - _raw | table * | outputlookup provider_lookup.csv </CODE></PRE> <P>It created a 2.2 GBs file on the SH's file system and I can see it under the 'Lookup table files' as -<BR /> <CODE>/opt/splunk/splunk/etc/apps/search/lookups/provider_lookup.csv</CODE> </P> <P>What should I do now?</P> Tue, 19 Apr 2016 01:50:36 GMT https://community.splunk.com/t5/Splunk-Search/How-to-store-a-large-search-result-set-into-a-lookup-table/m-p/215974#M63343 ddrillic 2016-04-19T01:50:36Z https://community.splunk.com/t5/Splunk-Search/How-to-store-a-large-search-result-set-into-a-lookup-table/m-p/215975#M63344 <P>You can use it with either <A href="http://docs.splunk.com/Documentation/Splunk/latest/SearchReference/Inputlookup">inputlookup</A> or just <A href="http://docs.splunk.com/Documentation/Splunk/latest/SearchReference/Lookup">lookup</A>.<BR /> Keep in mind you will need to create a lookup in order to do that.<BR /> Take a look at the very detailed <A href="http://docs.splunk.com/Documentation/Splunk/6.4.0/Knowledge/ConfigureCSVlookups">documentation</A> about this topic.</P> <P>Hope that helps.</P> Tue, 19 Apr 2016 09:13:34 GMT https://community.splunk.com/t5/Splunk-Search/How-to-store-a-large-search-result-set-into-a-lookup-table/m-p/215975#M63344 javiergn 2016-04-19T09:13:34Z https://community.splunk.com/t5/Splunk-Search/How-to-store-a-large-search-result-set-into-a-lookup-table/m-p/215976#M63345 <P>also be careful about saving large lookup files on your search head. You may bump into issues with syncing your bundle across indexers if you hit your maximum bundle size. You may need to whitelist the lookup folder within that app &gt; <A href="https://answers.splunk.com/answers/3436/how-could-i-optimize-distributed-replication-of-large-lookup-tables.html">https://answers.splunk.com/answers/3436/how-could-i-optimize-distributed-replication-of-large-lookup-tables.html</A></P> Tue, 19 Apr 2016 13:05:55 GMT https://community.splunk.com/t5/Splunk-Search/How-to-store-a-large-search-result-set-into-a-lookup-table/m-p/215976#M63345 rusty009 2016-04-19T13:05:55Z https://community.splunk.com/t5/Splunk-Search/How-to-store-a-large-search-result-set-into-a-lookup-table/m-p/215977#M63346 <P>Thanks a lot!</P> Tue, 19 Apr 2016 17:58:03 GMT https://community.splunk.com/t5/Splunk-Search/How-to-store-a-large-search-result-set-into-a-lookup-table/m-p/215977#M63346 ddrillic 2016-04-19T17:58:03Z https://community.splunk.com/t5/Splunk-Search/How-to-store-a-large-search-result-set-into-a-lookup-table/m-p/215978#M63347 <P>Thank you javiergn - worked nicely but slow with a lookup table of 2 GBs and physical memory of 16 MBs.</P> <P>The lookup part of the command looks like - <BR /> | lookup provider_lookup tin as prov_tin OUTPUT adr_ln_1_txt</P> Tue, 29 Sep 2020 09:27:14 GMT https://community.splunk.com/t5/Splunk-Search/How-to-store-a-large-search-result-set-into-a-lookup-table/m-p/215978#M63347 ddrillic 2020-09-29T09:27:14Z