As a csv lookup file starts getting more and more entries it's recommended to move the csv entries to a
Have a look here, it's a great document explaining why use a kvstore :
Some of the advantage described there for
csv are :
- Enables per-record insert/updates ("upserts"). - Allows optional data type enforcement on write operations. - Allows you to define field accelerations to improve search performance. - Provides REST API access to the data collection.
Also it's fairly easy to configure and use, in case you haven't done so before you can follow this guide :
@badoomi, optimizing lookup search may not be straight-forward without knowing your SPL and Splunk Infra ( as to how many Indexers you have got). However you can refer to following Splunk Documentation for one of tip to optimize lookup
By default lookup command runs with argument local=true which means it is executed on Search Peer not on Search Head. If you have multiple indexers and your SPL till the lookup command have only streaming commands then there would be an advantage of this otherwise not.
In essence you would need to test out
stats first then lookup vs
lookup first and stats next.
Do share your current SPL for community members to assist you better with your use case.