Dashboards & Visualizations

How to automatically refresh panel with "Search not executed: The maximum number of concurrent ... searches..." message

splunkuserCA1
Path Finder

I have an autogenerated dashboard with 160 panels. The good news: Each panel's search uses an accelerated saved search. The bad news: I have so many panels that I frequently see this message in about 40 of the panels:

Search not executed: The maximum number of concurrent historical searches on this instance has been reached., concurrency_category="historical", concurrency_context="instance-wide", current_concurrency=50, concurrency_limit=50
Labels (3)
0 Karma

acharlieh
Influencer

Out of curiosity, what is the use case for 160 different searches kicking off on a single dashboard on load? As a general rule of thumb that seems like either too much data to reason about at a glance on a single screen, poorly designed searches that are retrieving too little data from overlapping sources, or both.

Arguably if your SH and indexers are currently sized for 50 concurrent searches, the super expensive method would just mean tripling the number of CPU cores on the SH and each indexer. (Likely some memory and disk upgrades too...) obviously that scale up method incurs exponentially growing costs as you go.

So how do we do better with limited resources?

1) base searches with post processing. (You can have a single search that drives results across multiple panels)

2) referencing already executed Scheduled searches as base searches, especially if up-to-the-minute on dashboard load is not a requirement. (Accelerated searches are good for certain use cases, but if time windows are too soon, acceleration hasn’t run yet... Accelerated data models and Summary Indexing are also potential options for certain search patterns. )

3) think about your users and design your workflow of the dashboard... what are they looking for first? Why? What panels are only needed under specific conditions and otherwise ignored? would drill downs make sense to offload panels to other views based on the expected user workflow?

0 Karma

splunkuserCA1
Path Finder

As for my use case: I have 80 metrics and 2 graphs per metric. One search is a timechart, the other search is a stats to get the overall P90. I can't identify a way to combine the two and keep the search accelerated. The metric corresponds to the performance of a single URL (so 80 URLs). The interested parties want to be able to view this on one screen, concisely.

The URLs come from web server logs. Would a better designed base search pull in all the URLs and then have other searches that filter on URLs? But that way doesn't seem any more optimized than using the index as is.

 

Although I am a Splunk novice (having last used Splunk about a decade ago), Splunk is the one tool that has made me change my behavior to fit how Splunk works. In my opinion, it's a glaring gap in the product (and bad customer experience) when the user needs to know so much about how the underlying system works. Though it is still better than a decade ago!

 

 

0 Karma

splunkuserCA1
Path Finder

I think I figured it out. Please let me know if there is a better solution or way to do it, as I'm not a JavaScript expert.

 

In the JavaScript Console, this one-liner will find all panels with errors (class: alert alert-error), iterate over those objects to get a common parent node (class: dashboard-element single),  look for the Refresh button under the parent (expecting exactly one button), and click() it:

document.querySelectorAll(".alert.alert-error").forEach(function(item) {
  item.closest(".dashboard-element.single").querySelectorAll(".btn-pill.refresh")[0].click();
});

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...