Archive

Why is my scheduled search so much quicker than my adhoc search?

Champion

Hi,

I have a number of scheduled searches which run significantly faster than the same search run from the search-bar. I have no idea why this would happen - are there some settings that might cause this? The scheduled search is about 30-40 seconds (runs every 30 minutes throughout the day). The same adhoc search runs for minutes.

Search:

index=raytheon_proxy dstBytes>0
|eval totbytes=bytes/1024/1024/1024
|chart sum(totbytes)
0 Karma
1 Solution

SplunkTrust
SplunkTrust

First, try running the search with these settings -

1) The same time boundaries (probably earliest=@d latest=@m or earliest=-30m@m latest=@m)
2) set to fast, NOT set verbose.


Second, take a look at the job inspector for each job and see what is different.


Third, do this and see if anything changes. (It shouldn't, since the optimization should take care of that)

 index=raytheon_proxy dstBytes>0
 | fields bytes
 | eval totbytes=bytes/1024/1024/1024
 | chart sum(totbytes)

Fourth, see if you can get the same information from tstats, which would save massive amounts of search time. That would look something like this...

 | tstats sum(bytes) as bytes where index=raytheon_proxy dstBytes>0
 | eval totbytes=round(bytes/1024/1024/1024,2)

Fifth, not sure why you are testing dstBytes and summing bytes, but hopefully you do.

View solution in original post

SplunkTrust
SplunkTrust

First, try running the search with these settings -

1) The same time boundaries (probably earliest=@d latest=@m or earliest=-30m@m latest=@m)
2) set to fast, NOT set verbose.


Second, take a look at the job inspector for each job and see what is different.


Third, do this and see if anything changes. (It shouldn't, since the optimization should take care of that)

 index=raytheon_proxy dstBytes>0
 | fields bytes
 | eval totbytes=bytes/1024/1024/1024
 | chart sum(totbytes)

Fourth, see if you can get the same information from tstats, which would save massive amounts of search time. That would look something like this...

 | tstats sum(bytes) as bytes where index=raytheon_proxy dstBytes>0
 | eval totbytes=round(bytes/1024/1024/1024,2)

Fifth, not sure why you are testing dstBytes and summing bytes, but hopefully you do.

View solution in original post

Champion

Thanks. We are running 6.5, so the optimization isn't available yet. The third option was huge, should have realized it. The fourth, unfortunately is not available.

0 Karma

Ultra Champion

@DalJeanis with the clutch search tips as always!

I can't keep my mouth shut so here's another detail:

index=raytheon_proxy dstBytes>0
| stats sum(dstBytes) AS bytes
| eval TBs = bytes / pow( 1024 , 3 )

The point being that the search will be faster if the eval has less fields to compute on (such as after a stats transformation) and I find the pow easier to read then debugging vs having to count the # of times I wrote 1024.

BUT, this is tangential relative to why the search speeds are so different. Like was mentioned, check Job Inspector!

SplunkTrust
SplunkTrust

Nice point. Dividing three separate times by 1024 is inefficient, isn't it? If there were more than one event at that point - for instance, if we were calculating MB on the individual events for some reason - then it could save significant CPU cycles by multiplying pow(1024,3) out into a constant...

0 Karma

Ultra Champion

Oh, actually I was more emphasizing the use of stats and then eval.

Here's an example: we have one million events and we use stats to get it down to ten results. If we do eval then stats then we're calculating eval on a million events. If we flip it, we'll be doing eval on ten events. Furthermore, stats is a reducing command, so the payload sent back from the indexers should be summarized to just what's needed. eval is a streaming command and will obligate the indexers to send back full payload fidelity of the events and then make the search head use more memory to retain those events for the eval, only after which the stats happens (on the search heads).

Hopefully that clears up confusion I introduced.

SplunkTrust
SplunkTrust

@sloshburch - Got it. One slight adjustment to wording ..., the eval before a stats is streamable, so it is a million calculations done at the indexers, not at the search head. Nonetheless, with those ratios, you want it done at the search head. If the average ratio in your data were, say, five to one, and you had 50 indexers, then you'd want it done at the indexers.

0 Karma

Ultra Champion

Great point! Thanks for correcting me. Def not a typo but a full on mistake by me. Great correction and thanks!

0 Karma