Why and when is Splunk differing between canceling and queuing searches?

[HeinzWaescher]

Hi,

I'm wondering why (and when) there is a different handling when a lot of searches are running at the same time

• List item waiting for your queued job

vs.

• List item "Search not executed: The maximum number of concurrent historical searches on this instance has been reached."

For a dashboard user option 1 is much better, because he will see the needed results in the end. When option 2 is used, the dashboard will not complete

[gcusello]

Hi HeinzWaescher,
usually searches are queued when they reach the maximum number of concurrent historical searches.

The best way to solve the problem is possibly upgrade your hardware (CPUs) and modifying limits.conf.

At the same time it could be a good idea (to use everytime) to try to optimize your dashboard using post processing search (see https://docs.splunk.com/Documentation/Splunk/latest/Viz/Savedsearches ), you can find an example in Splunk 6.x Dashboard Examples app.

Bye.
Giuseppe

[HeinzWaescher]

Hi cusello,

thanks for your reply. We know about the option to modify hardware & configs. But the issue would not be such a problem, when all searches would be queued. Then it would simply take longer to load the dashboard.
But we often see these canceled search, which finally means that the dashboard won't finalze. But I haven't found a pattern when it happens.

[gcusello]

Hi HeinzWaescher,
Maybe some of your searches are timeouted.
if you open a search in the Search dashboard can you run it?
after result, see job inspector and see if meybe there is a timeout.
Bye.
Giuseppe

[HeinzWaescher]

I can't click the "open in search" button for these searches. The job Inspector says "unknown sid"

[gcusello]

open in search before they are timeouted.
Bye.
Giuseppe

[HeinzWaescher]

The error appears instantly when the dashboard is opened and Splunk tries to run to many searches. So there is no time until a timeout so that I could crosscheck it.
(Or I just don't understand what you mean :))

[gcusello]

take the search from your dashboard source and execute it in search (manually inserting eventual parameters) , to see if the problem is a timeout.
Bye.
Giuseppe

[HeinzWaescher]

ah okay 🙂 that works fine, starting instantly and finalizing fast.

[gcusello]

Try to rebuild your dashboard search by search and then putting inputs, maybe there's an error in parameters passing.
Bye.
Giuseppe

[HeinzWaescher]

I think I got it.
I created two dashboards. The first included saved non-scheduled searches, the second includes the same searches as inline searches.

I opened both dashboards:
The first dashboard started calculating results for max limit of concurrent searches, the rest was cancelled.
In the second dashboard all inline searches were queued and finalized step by step, nothing was cancelled. So it depends how the search in implemented in the dashboard.