Many Users Running Dashboards Simultaneously Causes High Indexer CPU

[jordanking1992]

Hello All,

Our environment consists of an indexer cluster scaled for 1 TB of data per day. On average, we have about 30 users logged running ad-hoc searches and about 40 scheduled searches running along side those queries. For 360 days of the year, the average CPU of our indexer cluster is no higher than 25%. But for 1 week of the year during the thanksgiving time range, we have about 65 users logged in, running ad-hoc queries, and loading multiple dashboards to monitor sales data during this time of the year. During this week of the year, the CPU on our indexers stays consistently at 90%-100% which we have attributed to many users loading dashboards with many panels simultaneously along with normal ad-hoc and scheduled searching.

My question is, what recommendations are our there for combating this increased usage and prevent the CPU from being pegged at 100% for 1 week of the year? We are thinking about limiting the amount of searches each user is allowed to run concurrently but fear that many users will complain that their searches are queued.

Any suggestions are much appreciated.

Respectfully,

[richgalloway]

You have a few options.

Add more SHs to the cluster during the peak period.  This is easier if the SHC uses VMs.

If the users are in one room, put some of the dashboards on monitors on the wall so many people can view a single instance of the dashboards.

Make the dashboards more efficient by making the searches better or by using base searches with post-processing or by moving the searches out of the dashboards into scheduled searches.  In the last option, the dashboard will load the most recent results of the search rather than triggering a new search.  This has an additional advantage of all users seeing the same values on the dashboard.

Limiting the searches each user can run.  This may be counter-productive so save it as the last resort.

[jordanking1992]

Hey richgalloway,

Appreciate the recommendations. Can you elaborate more on how adding more SH's will reduce the indexers CPU usage? We currently have a 3 node search head cluster tied to a 11 node indexer cluster.

[richgalloway]

Oops!  Disregard that.  I saw "cluster" and read "search head cluster".  

[jordanking1992]

No worries. Thanks again. Will look into post-processing of dashboard panels.

Thanks!