Splunk Search

index vs lookup

johnsasikumar
Path Finder

Hello All,

Am trying to optimize the performance of a dashboard that was built some time back. The existing dashboard has been created by joining atleast 10 lookup files in the same query for a panel. And over time the lookup has increased in size going more than 1,00,000 rows. This has caused a lot of problems in the join conditions made in the query.

  1. I would like to understand what is the search performance differnce when data is from a lookup or loaded from index.

  2. Does lookup command have limitations like the join command like, what is the max limit for a lookup command. Can it be used instead of join when data is from a lookup.

  3. Is there a difference in join limit set in limits.conf for data from Index and data from lookup. I have a scenario where the limits.conf default value for join and subquery has been increased and also am using max=0 in my join. But the results are not coming as expected. It works perfectly when i optimize the subquery having lookup to have less than 50,000 rows.

Any thoughts or advise on this.

0 Karma

codebuilder
SplunkTrust
SplunkTrust

This is a great opportunity to implement a accelerated datamodel or use KV Store, and move away from lookups.

Lookups are not indexed and become slower as their size increases. Think of the performance difference between a plain SQL query and a stored procedure, same logic applies.

----
An upvote would be appreciated and Accept Solution if it helps!
0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...