topic Re: Why and when is Splunk differing between canceling and queuing searches? in Splunk Search

Why and when is Splunk differing between canceling and queuing searches?

HeinzWaescher — Fri, 04 Aug 2017 08:08:53 GMT

Hi,

I'm wondering why (and when) there is a different handling when a lot of searches are running at the same time

List item waiting for your queued job

vs.

List item "Search not executed: The maximum number of concurrent historical searches on this instance has been reached."

For a dashboard user option 1 is much better, because he will see the needed results in the end. When option 2 is used, the dashboard will not complete

Re: Why and when is Splunk differing between canceling and queuing searches?

gcusello — Fri, 04 Aug 2017 08:16:10 GMT

Hi HeinzWaescher,
usually searches are queued when they reach the maximum number of concurrent historical searches.

The best way to solve the problem is possibly upgrade your hardware (CPUs) and modifying limits.conf.

At the same time it could be a good idea (to use everytime) to try to optimize your dashboard using post processing search (see https://docs.splunk.com/Documentation/Splunk/latest/Viz/Savedsearches ), you can find an example in Splunk 6.x Dashboard Examples app.

Bye.
Giuseppe

Re: Why and when is Splunk differing between canceling and queuing searches?

HeinzWaescher — Fri, 04 Aug 2017 08:23:57 GMT

Hi cusello,

thanks for your reply. We know about the option to modify hardware & configs. But the issue would not be such a problem, when all searches would be queued. Then it would simply take longer to load the dashboard.
But we often see these canceled search, which finally means that the dashboard won't finalze. But I haven't found a pattern when it happens.

Re: Why and when is Splunk differing between canceling and queuing searches?

gcusello — Fri, 04 Aug 2017 09:44:30 GMT

Hi HeinzWaescher,
Maybe some of your searches are timeouted.
if you open a search in the Search dashboard can you run it?
after result, see job inspector and see if meybe there is a timeout.
Bye.
Giuseppe

Re: Why and when is Splunk differing between canceling and queuing searches?

HeinzWaescher — Fri, 04 Aug 2017 13:02:13 GMT

I can't click the "open in search" button for these searches. The job Inspector says "unknown sid"

Re: Why and when is Splunk differing between canceling and queuing searches?

gcusello — Fri, 04 Aug 2017 13:27:40 GMT

open in search before they are timeouted.
Bye.
Giuseppe

Re: Why and when is Splunk differing between canceling and queuing searches?

HeinzWaescher — Fri, 04 Aug 2017 13:39:50 GMT

The error appears instantly when the dashboard is opened and Splunk tries to run to many searches. So there is no time until a timeout so that I could crosscheck it.
(Or I just don't understand what you mean :))

Re: Why and when is Splunk differing between canceling and queuing searches?

gcusello — Fri, 04 Aug 2017 13:59:13 GMT

take the search from your dashboard source and execute it in search (manually inserting eventual parameters) , to see if the problem is a timeout.
Bye.
Giuseppe

Re: Why and when is Splunk differing between canceling and queuing searches?

HeinzWaescher — Fri, 04 Aug 2017 14:04:21 GMT

ah okay 🙂 that works fine, starting instantly and finalizing fast.

Re: Why and when is Splunk differing between canceling and queuing searches?

gcusello — Fri, 04 Aug 2017 14:10:16 GMT

Try to rebuild your dashboard search by search and then putting inputs, maybe there's an error in parameters passing.
Bye.
Giuseppe

Re: Why and when is Splunk differing between canceling and queuing searches?

HeinzWaescher — Fri, 04 Aug 2017 14:28:09 GMT

I think I got it.
I created two dashboards. The first included saved non-scheduled searches, the second includes the same searches as inline searches.

I opened both dashboards:
The first dashboard started calculating results for max limit of concurrent searches, the rest was cancelled.
In the second dashboard all inline searches were queued and finalized step by step, nothing was cancelled. So it depends how the search in implemented in the dashboard.