Dashboards & Visualizations

Why do saved searches take too much time in dashboards?

Javo222
Path Finder

I have several dashboards with 8 to 15 graphs/saved searches each and when I try to display them I get "The maximum number of historical concurrent system-wide searches has been reached. current=6 maximum=6 SearchId=admin_admin_search..." I understand why and I tried increasing base_max_searches = 2 but my CPU maxes out at 100% and my PC freezes.

What I would like to do is to run maybe 6 concurrent searches and when these are do, run the next batch of 6.

Other solution I thought of could be to used cache data/searches but I wonder about the following points:
- How can I cache a saved search so it's not executed each time I open the dashboard?
- I stop my Splunk service whenever I'm done with looking at the dashboard. Some weeks later, I will need to look at them again and will start Splunk. Can the saved searches be preserved?
- My saved searches have a cron schedule in savedsearches.conf, similar to cron_schedule = 30 20 1-31 * * with different time of the day.

I barely need to refresh my data, so I don't mind having to use the data cached even if it's 1 month old...

Which parameter can allow me to implement one of these solutions? Or can you think of a better way?

I got this Splunk setup from someone else so I'm not exactly sure about the entire configuration.

I'm using Splunk Enterprise 6.3.0

0 Karma

woodcock
Esteemed Legend

Are you running any realtime searches? If so, that is your entire problem. DO NOT use realtime, especially in a dashboard.

0 Karma

cmeerbeek
Path Finder

This related post has a good answer which points you in the right direction:
https://answers.splunk.com/answers/233092/how-to-cache-app-dashboard-to-avoid-search-query-e.html

You need to use scheduled searches and call those reports. Splunk will first try to get the cached results and will not execute the search.

If you search for "Splunk dashboard saved searches" or "Splunk dashboard cache" you will find other topics about this matter.

Get Updates on the Splunk Community!

Splunk Classroom Chronicles: Training Tales and Testimonials

Welcome to the "Splunk Classroom Chronicles" series, created to help curious, career-minded learners get ...

Access Tokens Page - New & Improved

Splunk Observability Cloud recently launched an improved design for the access tokens page for better ...

Stay Connected: Your Guide to November Tech Talks, Office Hours, and Webinars!

🍂 Fall into November with a fresh lineup of Community Office Hours, Tech Talks, and Webinars we’ve ...