https://community.splunk.com/t5/Splunk-Search/Lookup-table-performance-question/m-p/24606#M4564 <P>There aren't any absolutes, but:</P> <UL> <LI>if you have a large lookup but only need to reference discrete parts of the lookup in discrete, non-overlapping searches, then it would be fine to break them into several smaller lookups</LI> <LI>if you have several lookups and you intend to frequently reference more than one of them at a time (either manually via the search language or automatically via props.conf), you would be best served to combine them in to one larger lookup.</LI> <LI>30 MB lookups are not too big by any stretch, but if they change constantly and/or you have a large distributed environment, you might start to experience poor performance without some tuning</LI> <LI>when a given lookup table gets rather large (more than a few hundred MB), the best practice is to move to external lookups (usually leveraging a script that queries a database or binary-tree) rather than to split the lookups into several smaller lookups</LI> </UL> Wed, 12 Jan 2011 05:17:00 GMT araitz 2011-01-12T05:17:00Z https://community.splunk.com/t5/Splunk-Search/Lookup-table-performance-question/m-p/24605#M4563 <P>I am experimenting with some searches that will need to do lookups on some fairly big tables (30 MB or more). I'm wondering whether it will be faster for Splunk to do a single lookup on a really large table or if I should just chain together several lookups on smaller tables. And I'm curious how big a table can get before it should be broken down into smaller sequential lookups (if ever).</P> <P>Thx.</P> Wed, 12 Jan 2011 00:04:44 GMT https://community.splunk.com/t5/Splunk-Search/Lookup-table-performance-question/m-p/24605#M4563 jambajuice 2011-01-12T00:04:44Z https://community.splunk.com/t5/Splunk-Search/Lookup-table-performance-question/m-p/24606#M4564 <P>There aren't any absolutes, but:</P> <UL> <LI>if you have a large lookup but only need to reference discrete parts of the lookup in discrete, non-overlapping searches, then it would be fine to break them into several smaller lookups</LI> <LI>if you have several lookups and you intend to frequently reference more than one of them at a time (either manually via the search language or automatically via props.conf), you would be best served to combine them in to one larger lookup.</LI> <LI>30 MB lookups are not too big by any stretch, but if they change constantly and/or you have a large distributed environment, you might start to experience poor performance without some tuning</LI> <LI>when a given lookup table gets rather large (more than a few hundred MB), the best practice is to move to external lookups (usually leveraging a script that queries a database or binary-tree) rather than to split the lookups into several smaller lookups</LI> </UL> Wed, 12 Jan 2011 05:17:00 GMT https://community.splunk.com/t5/Splunk-Search/Lookup-table-performance-question/m-p/24606#M4564 araitz 2011-01-12T05:17:00Z