Saved Search Issue--Very Slow Dispatch and Get Results

[ks2211]

Hi,

I have a saved search set up in Splunk.

Using the REST API, when I dispatch that saved search and then get the results back, it takes ~40-60 seconds for this particular saved search (I have a few others that complete in 2-3 seconds to dispatch and fetch results).

The saved search query looks something like this (obfuscating variable names, index names, etc):

index=myIndex source=someSource | rename var1 AS VAR1 var2 AS VAR2 | fields - _raw | join type=inner VAR3 [search earliest=1 latest=now index=otherIndex |fields - _raw | fields + field1, field2| rename var4 AS VAR4] |fields + field3,field4| join type=left someID [|search index=thirdIndex | fields - _raw | fields + field5, field6 | dedup SOMEVAR ] | search someVAr=* | top VAR10 limit=10

So when I run this saved search in the UI itself and look at the activity log for jobs, I'll see my saved search execute BUT I also see several subsearches executing. My saved search doesn't start until the subsearches are 100% and done.

Why is this happening? Is there a way to set up my saved search to execute this full query in one go instead of doing multiple subsearches?

I looked at "loadjob" and this worked partially. For some of my searches, I use the $args$ to run based on some criterira. With the loadjob, I wasn't able to properly do a replace of the args with the value I wanted so this wouldn't work.

thanks!

[martin_mueller]

It's intended and documented behaviour to let subsearches finish before starting the main search.

What's not quite intended is to do convoluted chains of join, also with dedup on top - usually these can be solved better and more splunky. However, it's quite hard to create a better alternative with a lot of obfuscation going on and without knowing the requirements for the search. Here's a good overview of the issue: https://answers.splunk.com/answers/129424/how-to-compare-fields-over-multiple-sourcetypes-without-jo...