Splunk Search

optimize lookup search

New Member

I have a lookup file with 50,000 records. When I want to do a search, it takes a lot of time to find my results. Is there a way to get faster and better searches result?

0 Karma

SplunkTrust
SplunkTrust

Hi @badoomi,

As a csv lookup file starts getting more and more entries it's recommended to move the csv entries to a kvstore.

Have a look here, it's a great document explaining why use a kvstore :
http://dev.splunk.com/view/webframework-developapps/SP-CAAAEY7
Some of the advantage described there for kvstore vs csv are :

- Enables per-record insert/updates
  ("upserts").
- Allows optional data type enforcement
  on write operations.
- Allows you to define field
  accelerations to improve search
  performance.
- Provides REST API access to the data
  collection.

Also it's fairly easy to configure and use, in case you haven't done so before you can follow this guide :
https://docs.splunk.com/Documentation/Splunk/7.3.0/Knowledge/ConfigureKVstorelookups

Cheers,
David

0 Karma

SplunkTrust
SplunkTrust

@badoomi, optimizing lookup search may not be straight-forward without knowing your SPL and Splunk Infra ( as to how many Indexers you have got). However you can refer to following Splunk Documentation for one of tip to optimize lookup

By default lookup command runs with argument local=true which means it is executed on Search Peer not on Search Head. If you have multiple indexers and your SPL till the lookup command have only streaming commands then there would be an advantage of this otherwise not.

In essence you would need to test out stats first then lookup vs lookup first and stats next.

Do share your current SPL for community members to assist you better with your use case.

____________________________________________
| makeresults | eval message= "Happy Splunking!!!"

New Member

i have one indexer and one forwarder.i create a automatic lookup.my search is
index=fw or index=waf | where ip=m_ip | stats count by src,category

0 Karma

Path Finder

Can you give an example of the search you are attempting on the lookup.

ie | lookup or | inputlookup

0 Karma