Deployment Monitor and High CPU

[jonathanmorcom]

This version is causing very high cpu usage on our Deployment Server when trying to visit any components of the App especially the home page...

It's very slow and doesn't completely display all the information like the old version.

I've followed the advice here: http://docs.splunk.com/Documentation/DepMon/latest/DeployDepMon/PopulatetheDeploymentMonitorwithhist... and reduce the 4 listed Report Acceleration Summaries to 1 day but the app is still basically un-usable as it stands.

Any suggestions to troubleshoot / improve this?

[mkinsley_splunk · Accepted solution]

Setting your Acceleration Summaries to last 1 day will destroy performance. The setting for duration is meant to help manage the maximum space used on disk by the summaries. With Report Acceleration, if summaries are not available, the engine falls back to running the full search, so if you set your summaries to live for 1 day, you will basically be forcing your searches to run in brute force mode.

The initial creation of the Report Summaries does take some time. Once they are generated you will see much improved performance in reporting.

Report Acceleration Summary status can be viewed from :

Manager > Report Acceleration Summaries

[jonathanmorcom]

I did a complete re-deployment of this onto a new VM and had the same issues with a vanilla install, no changes to summary index retention. I Did see our summary indexes fill with data which took a long time due to the size of our environment. But search performance for all the standard dashboard reports took hours to complete and show all the results.

In the end I've had to roll back to Dep Mon 4.3.5 which does work correctly.

I've raised a ticket with Splunk so hopefully it's fixed in next release, as it appears unable to handle large deployments right now.

[marcoscala]

I installed it on our test/training server, with not much data (I have to admin), but I haven't seen those heavey CPU Usage. We have CentOS VM with 4 virtual Core.