Deployment Architecture

Deployment Monitor and High CPU

jonathanmorcom
Explorer

This version is causing very high cpu usage on our Deployment Server when trying to visit any components of the App especially the home page...

It's very slow and doesn't completely display all the information like the old version.

I've followed the advice here: http://docs.splunk.com/Documentation/DepMon/latest/DeployDepMon/PopulatetheDeploymentMonitorwithhist... and reduce the 4 listed Report Acceleration Summaries to 1 day but the app is still basically un-usable as it stands.

Any suggestions to troubleshoot / improve this?

0 Karma
1 Solution

mkinsley_splunk
Splunk Employee
Splunk Employee

Setting your Acceleration Summaries to last 1 day will destroy performance. The setting for duration is meant to help manage the maximum space used on disk by the summaries. With Report Acceleration, if summaries are not available, the engine falls back to running the full search, so if you set your summaries to live for 1 day, you will basically be forcing your searches to run in brute force mode.

The initial creation of the Report Summaries does take some time. Once they are generated you will see much improved performance in reporting.

Report Acceleration Summary status can be viewed from :

Manager > Report Acceleration Summaries

View solution in original post

mkinsley_splunk
Splunk Employee
Splunk Employee

Setting your Acceleration Summaries to last 1 day will destroy performance. The setting for duration is meant to help manage the maximum space used on disk by the summaries. With Report Acceleration, if summaries are not available, the engine falls back to running the full search, so if you set your summaries to live for 1 day, you will basically be forcing your searches to run in brute force mode.

The initial creation of the Report Summaries does take some time. Once they are generated you will see much improved performance in reporting.

Report Acceleration Summary status can be viewed from :

Manager > Report Acceleration Summaries

jonathanmorcom
Explorer

I did a complete re-deployment of this onto a new VM and had the same issues with a vanilla install, no changes to summary index retention. I Did see our summary indexes fill with data which took a long time due to the size of our environment. But search performance for all the standard dashboard reports took hours to complete and show all the results.

In the end I've had to roll back to Dep Mon 4.3.5 which does work correctly.

I've raised a ticket with Splunk so hopefully it's fixed in next release, as it appears unable to handle large deployments right now.

0 Karma

marcoscala
Builder

I installed it on our test/training server, with not much data (I have to admin), but I haven't seen those heavey CPU Usage. We have CentOS VM with 4 virtual Core.

0 Karma
Get Updates on the Splunk Community!

Improve Your Security Posture

Watch NowImprove Your Security PostureCustomers are at the center of everything we do at Splunk and security ...

Maximize the Value from Microsoft Defender with Splunk

 Watch NowJoin Splunk and Sens Consulting for this Security Edition Tech TalkWho should attend:  Security ...

This Week's Community Digest - Splunk Community Happenings [6.27.22]

Get the latest news and updates from the Splunk Community here! News From Splunk Answers ✍️ Splunk Answers is ...