Dashboards & Visualizations

Why do saved searches take too much time in dashboards?

Javo222
Path Finder

I have several dashboards with 8 to 15 graphs/saved searches each and when I try to display them I get "The maximum number of historical concurrent system-wide searches has been reached. current=6 maximum=6 SearchId=admin_admin_search..." I understand why and I tried increasing base_max_searches = 2 but my CPU maxes out at 100% and my PC freezes.

What I would like to do is to run maybe 6 concurrent searches and when these are do, run the next batch of 6.

Other solution I thought of could be to used cache data/searches but I wonder about the following points:
- How can I cache a saved search so it's not executed each time I open the dashboard?
- I stop my Splunk service whenever I'm done with looking at the dashboard. Some weeks later, I will need to look at them again and will start Splunk. Can the saved searches be preserved?
- My saved searches have a cron schedule in savedsearches.conf, similar to cron_schedule = 30 20 1-31 * * with different time of the day.

I barely need to refresh my data, so I don't mind having to use the data cached even if it's 1 month old...

Which parameter can allow me to implement one of these solutions? Or can you think of a better way?

I got this Splunk setup from someone else so I'm not exactly sure about the entire configuration.

I'm using Splunk Enterprise 6.3.0

0 Karma

woodcock
Esteemed Legend

Are you running any realtime searches? If so, that is your entire problem. DO NOT use realtime, especially in a dashboard.

0 Karma

cmeerbeek
Path Finder

This related post has a good answer which points you in the right direction:
https://answers.splunk.com/answers/233092/how-to-cache-app-dashboard-to-avoid-search-query-e.html

You need to use scheduled searches and call those reports. Splunk will first try to get the cached results and will not execute the search.

If you search for "Splunk dashboard saved searches" or "Splunk dashboard cache" you will find other topics about this matter.

Get Updates on the Splunk Community!

Continuing Innovation & New Integrations Unlock Full Stack Observability For Your ...

You’ve probably heard the latest about AppDynamics joining the Splunk Observability portfolio, deepening our ...

Monitoring Amazon Elastic Kubernetes Service (EKS)

As we’ve seen, integrating Kubernetes environments with Splunk Observability Cloud is a quick and easy way to ...

Cloud Platform & Enterprise: Classic Dashboard Export Feature Deprecation

As of Splunk Cloud Platform 9.3.2408 and Splunk Enterprise 9.4, classic dashboard export features are now ...