How to automatically refresh panel with "Search not executed: The maximum number of concurrent ... searches..." message

[splunkuserCA1]

I have an autogenerated dashboard with 160 panels. The good news: Each panel's search uses an accelerated saved search. The bad news: I have so many panels that I frequently see this message in about 40 of the panels:

Search not executed: The maximum number of concurrent historical searches on this instance has been reached., concurrency_category="historical", concurrency_context="instance-wide", current_concurrency=50, concurrency_limit=50

If I hover over those panels and click the "Refresh" icons, after the other searches have finished, the search tries again and succeeds. I see the information I was looking for in the panel.

I accept that there is a limit to concurrent searches. I'm willing to wait. But clicking the individual "Refresh" icon in each of panels with the "Search not executed" message is a lot of work.

Ideally, I'd like the search panel to automatically retry as capacity frees up, without me clicking the "Refresh" icon. Is that possible?

If it's not possible, is there a programmatic way for me to automatically kick off the "Refresh" icon? I tried using the JavaScript console with something like:  mvc.Components.get("baseSearch_item3_P1").startSearch(); but it couldn't resolve the mvc object. I was thinking I could walk through the DOM and execute an action for each panel that contains the text "Search not Executed", but I don't have the expertise to figure out how to do that and I can't even get the startSearch() action working for one individual search. Any tips or workarounds around automatically refreshing with one JavaScript command via the JavaScript console?

As a last ditch workaround, I've separated the dashboard into 6 sections with "depends" tokens that are controlled by an input dropdown. Each dropdown item loads 1/6th of the dashboard at a time. It's an ugly, hacky way that I really want to avoid. My coworkers will laugh at me for doing that.

[acharlieh]

Out of curiosity, what is the use case for 160 different searches kicking off on a single dashboard on load? As a general rule of thumb that seems like either too much data to reason about at a glance on a single screen, poorly designed searches that are retrieving too little data from overlapping sources, or both.

Arguably if your SH and indexers are currently sized for 50 concurrent searches, the super expensive method would just mean tripling the number of CPU cores on the SH and each indexer. (Likely some memory and disk upgrades too...) obviously that scale up method incurs exponentially growing costs as you go.

So how do we do better with limited resources?

1) base searches with post processing. (You can have a single search that drives results across multiple panels)

2) referencing already executed Scheduled searches as base searches, especially if up-to-the-minute on dashboard load is not a requirement. (Accelerated searches are good for certain use cases, but if time windows are too soon, acceleration hasn’t run yet... Accelerated data models and Summary Indexing are also potential options for certain search patterns. )

3) think about your users and design your workflow of the dashboard... what are they looking for first? Why? What panels are only needed under specific conditions and otherwise ignored? would drill downs make sense to offload panels to other views based on the expected user workflow?

[splunkuserCA1]

As for my use case: I have 80 metrics and 2 graphs per metric. One search is a timechart, the other search is a stats to get the overall P90. I can't identify a way to combine the two and keep the search accelerated. The metric corresponds to the performance of a single URL (so 80 URLs). The interested parties want to be able to view this on one screen, concisely.

The URLs come from web server logs. Would a better designed base search pull in all the URLs and then have other searches that filter on URLs? But that way doesn't seem any more optimized than using the index as is.

Although I am a Splunk novice (having last used Splunk about a decade ago), Splunk is the one tool that has made me change my behavior to fit how Splunk works. In my opinion, it's a glaring gap in the product (and bad customer experience) when the user needs to know so much about how the underlying system works. Though it is still better than a decade ago!

[splunkuserCA1]

I think I figured it out. Please let me know if there is a better solution or way to do it, as I'm not a JavaScript expert.

In the JavaScript Console, this one-liner will find all panels with errors (class: alert alert-error), iterate over those objects to get a common parent node (class: dashboard-element single),  look for the Refresh button under the parent (expecting exactly one button), and click() it:

document.querySelectorAll(".alert.alert-error").forEach(function(item) {
  item.closest(".dashboard-element.single").querySelectorAll(".btn-pill.refresh")[0].click();
});