Splunk performance issue

[uagraw01]

Hello Splunkers!!

A generic question I want to ask. There are 40+ dashboards in which customer are using any optimization in any dashboards. They are using direct index search across all the panels. They are not using any base search and summary index in any of the dashboard panels. Sometime in one dashboard they are using 60+ panels with all index searches. Could any help me to provide all the consequences will happen in this scenerio?

Thanks in advance

[bowesmana]

Consequences...

• Poor performance of your dashboard
• Poor performance for other users
• Excessive usage of an SVC licence if using SVC in Splunk Cloud - potentially causing additional licence costs to the organisation
• Skipped searches
• Your application will not be liked by others in your organisation
• Alerts may not fire and as such you may miss critical security detections the could indicate hackers are attacking your system, or that critical infrastructure is having performance issues, resulting in an outage of your primary web site.

These are some, but not all of the consequences. All will depend on what you are using Splunk for, but I hope you get the picture.

I've seen one such dashboard such as yours with 60 panels all on auto refresh, all searching the same data independently and that one dashboard, out of 1000 others, was using a significant proportion of the compute cost across the search head cluster.

[PickleRick]

The number of _defined_ dashboards doesn't matter much. In fact most Splunk installations have many dashboards which are never used and as such they cause no problem. On the other hand even one dashboard used in parallel by many users can hog resources causing performance isues.

In general, if you (try to) overbook your resources depending on your configuration you might end up with delayed/skipped searches (remember that ad-hoc searches have priority so if you have unlimited roles, you can cause scheduled searches to be delayed/skipped), searches terminated due to resource exhaustion or in some cases even causing Splunk process to be killed due to memory exhaustion. There are many things that can go wrong.