Saved Search Issue--Very Slow Dispatch and Get Results



I have a saved search set up in Splunk.

Using the REST API, when I dispatch that saved search and then get the results back, it takes ~40-60 seconds for this particular saved search (I have a few others that complete in 2-3 seconds to dispatch and fetch results).

The saved search query looks something like this (obfuscating variable names, index names, etc):

index=myIndex source=someSource | rename var1 AS VAR1 var2 AS VAR2 | fields - _raw | join type=inner VAR3 [search earliest=1 latest=now index=otherIndex |fields - _raw | fields + field1, field2| rename var4 AS VAR4] |fields + field3,field4| join type=left someID [|search index=thirdIndex | fields - _raw | fields + field5, field6 | dedup SOMEVAR ] | search someVAr=* | top VAR10 limit=10

So when I run this saved search in the UI itself and look at the activity log for jobs, I'll see my saved search execute BUT I also see several subsearches executing. My saved search doesn't start until the subsearches are 100% and done.

Why is this happening? Is there a way to set up my saved search to execute this full query in one go instead of doing multiple subsearches?

I looked at "loadjob" and this worked partially. For some of my searches, I use the $args$ to run based on some criterira. With the loadjob, I wasn't able to properly do a replace of the args with the value I wanted so this wouldn't work.


0 Karma

Re: Saved Search Issue--Very Slow Dispatch and Get Results


It's intended and documented behaviour to let subsearches finish before starting the main search.

What's not quite intended is to do convoluted chains of join, also with dedup on top - usually these can be solved better and more splunky. However, it's quite hard to create a better alternative with a lot of obfuscation going on and without knowing the requirements for the search. Here's a good overview of the issue:

0 Karma