Splunk Enterprise

Many Users Running Dashboards Simultaneously Causes High Indexer CPU

jordanking1992
Path Finder

Hello All,

Our environment consists of an indexer cluster scaled for 1 TB of data per day. On average, we have about 30 users logged running ad-hoc searches and about 40 scheduled searches running along side those queries. For 360 days of the year, the average CPU of our indexer cluster is no higher than 25%. But for 1 week of the year during the thanksgiving time range, we have about 65 users logged in, running ad-hoc queries, and loading multiple dashboards to monitor sales data during this time of the year. During this week of the year, the CPU on our indexers stays consistently at 90%-100% which we have attributed to many users loading dashboards with many panels simultaneously along with normal ad-hoc and scheduled searching.

My question is, what recommendations are our there for combating this increased usage and prevent the CPU from being pegged at 100% for 1 week of the year? We are thinking about limiting the amount of searches each user is allowed to run concurrently but fear that many users will complain that their searches are queued.

Any suggestions are much appreciated.

Respectfully,

Labels (3)
0 Karma

richgalloway
SplunkTrust
SplunkTrust

You have a few options.

Add more SHs to the cluster during the peak period.  This is easier if the SHC uses VMs.

If the users are in one room, put some of the dashboards on monitors on the wall so many people can view a single instance of the dashboards.

Make the dashboards more efficient by making the searches better or by using base searches with post-processing or by moving the searches out of the dashboards into scheduled searches.  In the last option, the dashboard will load the most recent results of the search rather than triggering a new search.  This has an additional advantage of all users seeing the same values on the dashboard.

Limiting the searches each user can run.  This may be counter-productive so save it as the last resort.

---
If this reply helps you, an upvote would be appreciated.

jordanking1992
Path Finder

Hey richgalloway,

 

Appreciate the recommendations. Can you elaborate more on how adding more SH's will reduce the indexers CPU usage? We currently have a 3 node search head cluster tied to a 11 node indexer cluster.

0 Karma

richgalloway
SplunkTrust
SplunkTrust

Oops!  Disregard that.  I saw "cluster" and read "search head cluster".  

---
If this reply helps you, an upvote would be appreciated.
0 Karma

jordanking1992
Path Finder

No worries. Thanks again. Will look into post-processing of dashboard panels.

Thanks!

0 Karma
Get Updates on the Splunk Community!

Improve Your Security Posture

Watch NowImprove Your Security PostureCustomers are at the center of everything we do at Splunk and security ...

Maximize the Value from Microsoft Defender with Splunk

 Watch NowJoin Splunk and Sens Consulting for this Security Edition Tech TalkWho should attend:  Security ...

This Week's Community Digest - Splunk Community Happenings [6.27.22]

Get the latest news and updates from the Splunk Community here! News From Splunk Answers ✍️ Splunk Answers is ...