Lookups load 500 times in one search

[misteraufziehvo]

Hi,

the environment uses 170 lookups and during one single search, they get loaded exactly 500 times each wich sums up to thousands of times in total and a search job total of 800mb as long as we type index=*. I know I know, you shouldn't do that, but our customer did and this obviously is a bug (selfmade?). Btw: All lookups are set to global.

What happens here? How can we change it?
Thank you 🙂

[DavidHourani]

Hi @misteraufziehvogel,

Do you mean that your sourcetypes are using automatic lookups and when you run index=* all lookups are running at the same time ?

If you're trying to avoid that make sure you set your search type to fast mode instead of verbose.

Cheers,
David

[misteraufziehvo]

Hi @DavidHourani,

changing the search type doesn't change anything unfortunately. But I got closer to the answer: We do have 506 Indizes and all Lookups get loaded for every single Index..... we will be starting a campaign for the employees: Specify the Index 🙂

[lakshman239]

I think you may have one more issue to look-at. Look-ups (even if they are global and automatic) should be normally restricted to one sourcetype (can't see why we would enable across sourcetypes). So, you may want to look at that as well. Also, for users, you can default 'indexes' to search in the roles definition. So, if anyone uses index=* , it will restrict them to default ones, as opposed to search across all indexes, till the education/campaign is successful.

[DavidHourani]

yes, also make also make sure you select the appropriate index filter per role to reduce the load and try to avoid automatic lookups unless they are vital for your sourcetype.

Have a look at the answer below could help you forge some searches that won't call the automated lookups :
https://answers.splunk.com/answers/113653/ignore-automatic-lookup-just-for-a-search.html
The again when you automate something it means you need to be applied every single time... ends up backfiring when you have so many lookups 🙂