Deployment Architecture

Deployment Monitor and High CPU

jonathanmorcom
Explorer

This version is causing very high cpu usage on our Deployment Server when trying to visit any components of the App especially the home page...

It's very slow and doesn't completely display all the information like the old version.

I've followed the advice here: http://docs.splunk.com/Documentation/DepMon/latest/DeployDepMon/PopulatetheDeploymentMonitorwithhist... and reduce the 4 listed Report Acceleration Summaries to 1 day but the app is still basically un-usable as it stands.

Any suggestions to troubleshoot / improve this?

0 Karma
1 Solution

mkinsley_splunk
Splunk Employee
Splunk Employee

Setting your Acceleration Summaries to last 1 day will destroy performance. The setting for duration is meant to help manage the maximum space used on disk by the summaries. With Report Acceleration, if summaries are not available, the engine falls back to running the full search, so if you set your summaries to live for 1 day, you will basically be forcing your searches to run in brute force mode.

The initial creation of the Report Summaries does take some time. Once they are generated you will see much improved performance in reporting.

Report Acceleration Summary status can be viewed from :

Manager > Report Acceleration Summaries

View solution in original post

mkinsley_splunk
Splunk Employee
Splunk Employee

Setting your Acceleration Summaries to last 1 day will destroy performance. The setting for duration is meant to help manage the maximum space used on disk by the summaries. With Report Acceleration, if summaries are not available, the engine falls back to running the full search, so if you set your summaries to live for 1 day, you will basically be forcing your searches to run in brute force mode.

The initial creation of the Report Summaries does take some time. Once they are generated you will see much improved performance in reporting.

Report Acceleration Summary status can be viewed from :

Manager > Report Acceleration Summaries

jonathanmorcom
Explorer

I did a complete re-deployment of this onto a new VM and had the same issues with a vanilla install, no changes to summary index retention. I Did see our summary indexes fill with data which took a long time due to the size of our environment. But search performance for all the standard dashboard reports took hours to complete and show all the results.

In the end I've had to roll back to Dep Mon 4.3.5 which does work correctly.

I've raised a ticket with Splunk so hopefully it's fixed in next release, as it appears unable to handle large deployments right now.

0 Karma

marcoscala
Builder

I installed it on our test/training server, with not much data (I have to admin), but I haven't seen those heavey CPU Usage. We have CentOS VM with 4 virtual Core.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...