optimize lookup search

[badoomi]

I have a lookup file with 50,000 records. When I want to do a search, it takes a lot of time to find my results. Is there a way to get faster and better searches result?

[DavidHourani]

Hi @badoomi,

As a csv lookup file starts getting more and more entries it's recommended to move the csv entries to a kvstore.

Have a look here, it's a great document explaining why use a kvstore :
http://dev.splunk.com/view/webframework-developapps/SP-CAAAEY7
Some of the advantage described there for kvstore vs csv are :

- Enables per-record insert/updates
  ("upserts").
- Allows optional data type enforcement
  on write operations.
- Allows you to define field
  accelerations to improve search
  performance.
- Provides REST API access to the data
  collection.

Also it's fairly easy to configure and use, in case you haven't done so before you can follow this guide :
https://docs.splunk.com/Documentation/Splunk/7.3.0/Knowledge/ConfigureKVstorelookups

Cheers,
David

[niketn]

@badoomi, optimizing lookup search may not be straight-forward without knowing your SPL and Splunk Infra ( as to how many Indexers you have got). However you can refer to following Splunk Documentation for one of tip to optimize lookup

By default lookup command runs with argument local=true which means it is executed on Search Peer not on Search Head. If you have multiple indexers and your SPL till the lookup command have only streaming commands then there would be an advantage of this otherwise not.

In essence you would need to test out stats first then lookup vs lookup first and stats next.

Do share your current SPL for community members to assist you better with your use case.

[m_pham]

---

@niketn wrote:

@badoomi, optimizing lookup search may not be straight-forward without knowing your SPL and Splunk Infra ( as to how many Indexers you have got). However you can refer to following Splunk Documentation for one of tip to optimize lookup

By default lookup command runs with argument local=true which means it is executed on Search Peer not on Search Head. If you have multiple indexers and your SPL till the lookup command have only streaming commands then there would be an advantage of this otherwise not.

In essence you would need to test out stats first then lookup vs lookup first and stats next.

Do share your current SPL for community members to assist you better with your use case.

---

I think there may have been a typo this this original answer as the lookup command has local=false set by default - source: https://docs.splunk.com/Documentation/Splunk/9.1.1/SearchReference/Lookup

local

Syntax: local=<bool>
Description: If local=true, forces the lookup to run on the search head and not on any remote peers.
Default: false

---

[badoomi]

i have one indexer and one forwarder.i create a automatic lookup.my search is
index=fw or index=waf | where ip=m_ip | stats count by src,category

[aromanauskas]

Can you give an example of the search you are attempting on the lookup.

ie | lookup or | inputlookup