Splunk Search

Lookups load 500 times in one search

New Member

Hi,

the environment uses 170 lookups and during one single search, they get loaded exactly 500 times each wich sums up to thousands of times in total and a search job total of 800mb as long as we type index=*. I know I know, you shouldn't do that, but our customer did and this obviously is a bug (selfmade?). Btw: All lookups are set to global.

What happens here? How can we change it?
Thank you 🙂

0 Karma

SplunkTrust
SplunkTrust

Hi @misteraufziehvogel,

Do you mean that your sourcetypes are using automatic lookups and when you run index=* all lookups are running at the same time ?

If you're trying to avoid that make sure you set your search type to fast mode instead of verbose.

Cheers,
David

0 Karma

New Member

Hi @DavidHourani,

changing the search type doesn't change anything unfortunately. But I got closer to the answer: We do have 506 Indizes and all Lookups get loaded for every single Index..... we will be starting a campaign for the employees: Specify the Index 🙂

0 Karma

SplunkTrust
SplunkTrust

I think you may have one more issue to look-at. Look-ups (even if they are global and automatic) should be normally restricted to one sourcetype (can't see why we would enable across sourcetypes). So, you may want to look at that as well. Also, for users, you can default 'indexes' to search in the roles definition. So, if anyone uses index=* , it will restrict them to default ones, as opposed to search across all indexes, till the education/campaign is successful.

SplunkTrust
SplunkTrust

yes, also make also make sure you select the appropriate index filter per role to reduce the load and try to avoid automatic lookups unless they are vital for your sourcetype.

Have a look at the answer below could help you forge some searches that won't call the automated lookups :
https://answers.splunk.com/answers/113653/ignore-automatic-lookup-just-for-a-search.html
The again when you automate something it means you need to be applied every single time... ends up backfiring when you have so many lookups 🙂

0 Karma