topic Re: Ignore automatic lookup just for a search in Splunk Enterprise Security

Ignore automatic lookup just for a search

marcoscala — Wed, 04 Dec 2013 11:32:15 GMT

Hi!

do you think if there's a way to say Splunk to ignore automatic lookups just for a search? I'm configuring some custom reports on a Splunk installation with ES and PCI Apps. Those apps do an intensive usage of automatic lookups which are fine, but that introduce some overhead. Running my custom report I see from Search Inspector that the most time (28secs out of 31secs)is spent in command.search.lookups.

Is there a way to say: for this seach ignore automatic lookups?

Thanks a lot!

Re: Ignore automatic lookup just for a search

lukejadamec — Wed, 04 Dec 2013 14:28:09 GMT

Automatic lookup are used when the search includes the lookup output. For example: If you have an input RETURNCODE and an output returncode_name, and you run a search like this:

search |table RETURNCODE

The cost for command.search.lookups will be zero.

If you run the search:

search |table returncode_name

The cost for command.search.lookups will have a value.

What you'll need to do is create your custom search so that it uses the inputs to the lookup, and not the outputs.

Re: Ignore automatic lookup just for a search

marcoscala — Fri, 13 Dec 2013 15:49:23 GMT

I'm sorry to contraddict your reply: in the Splunk for PCI app, for instance, everytime there's a host, src or dest field, the automatic lookups are fired to produce more infos from the assets.csv file. So the cost of in my case has always a quite significant value.

The search "sourcetype=vanguard" in the last 15m has the following costs:
1.959 events in 8,65 seconds.
5,021 command.search
4,049 command.search.lookups

Re: Ignore automatic lookup just for a search

lukejadamec — Fri, 13 Dec 2013 16:34:00 GMT

You can lead a horse to water....
Look, when you search for the sourcetype the search will of course find all fields and all associated look ups.

Like I said, you need to include in your search only fields that are not lookup fields.
Try this with your search and check the cost (this is very basic example):
sourcetype=vanguard |fields index

Re: Ignore automatic lookup just for a search

lukejadamec — Sat, 14 Dec 2013 01:17:20 GMT

If this answer helped, please mark it as excepted.

Re: Ignore automatic lookup just for a search

millern4 — Mon, 24 Aug 2015 11:16:26 GMT

Although this thread is quite old, I'd still like to see if anyone out there is still experiencing this exact same issue?

We've been working with support for some time now to troubleshoot poor search performance on our (4) ES SH Cluster where a majority of our time is spent in command.search.lookups, where in our (6) Non ES SH Cluster running the exact same search produces lighting fast results.

We are obviously aware of the nature of how ES functions with the automatic lookups, but I"m curious as others experience with Enterprise Security searching vs non ES Search performance? Is it similar, better, worse, and how close is the experience between the environments for end users?

Thanks

Re: Ignore automatic lookup just for a search

JimGat_SSI — Thu, 27 Oct 2016 14:50:47 GMT

Did you ever get an answer to this? I am wanting to disable the automatic lookups for specific searches using triggered fields.