Deployment Architecture

Deployment Monitor and High CPU

jonathanmorcom
Explorer

This version is causing very high cpu usage on our Deployment Server when trying to visit any components of the App especially the home page...

It's very slow and doesn't completely display all the information like the old version.

I've followed the advice here: http://docs.splunk.com/Documentation/DepMon/latest/DeployDepMon/PopulatetheDeploymentMonitorwithhist... and reduce the 4 listed Report Acceleration Summaries to 1 day but the app is still basically un-usable as it stands.

Any suggestions to troubleshoot / improve this?

0 Karma
1 Solution

mkinsley_splunk
Splunk Employee
Splunk Employee

Setting your Acceleration Summaries to last 1 day will destroy performance. The setting for duration is meant to help manage the maximum space used on disk by the summaries. With Report Acceleration, if summaries are not available, the engine falls back to running the full search, so if you set your summaries to live for 1 day, you will basically be forcing your searches to run in brute force mode.

The initial creation of the Report Summaries does take some time. Once they are generated you will see much improved performance in reporting.

Report Acceleration Summary status can be viewed from :

Manager > Report Acceleration Summaries

View solution in original post

mkinsley_splunk
Splunk Employee
Splunk Employee

Setting your Acceleration Summaries to last 1 day will destroy performance. The setting for duration is meant to help manage the maximum space used on disk by the summaries. With Report Acceleration, if summaries are not available, the engine falls back to running the full search, so if you set your summaries to live for 1 day, you will basically be forcing your searches to run in brute force mode.

The initial creation of the Report Summaries does take some time. Once they are generated you will see much improved performance in reporting.

Report Acceleration Summary status can be viewed from :

Manager > Report Acceleration Summaries

jonathanmorcom
Explorer

I did a complete re-deployment of this onto a new VM and had the same issues with a vanilla install, no changes to summary index retention. I Did see our summary indexes fill with data which took a long time due to the size of our environment. But search performance for all the standard dashboard reports took hours to complete and show all the results.

In the end I've had to roll back to Dep Mon 4.3.5 which does work correctly.

I've raised a ticket with Splunk so hopefully it's fixed in next release, as it appears unable to handle large deployments right now.

0 Karma

marcoscala
Builder

I installed it on our test/training server, with not much data (I have to admin), but I haven't seen those heavey CPU Usage. We have CentOS VM with 4 virtual Core.

0 Karma
Get Updates on the Splunk Community!

New Learning Videos on Topics Most Requested by You! Plus This Month’s New Splunk ...

Splunk Lantern is a customer success center that provides advice from Splunk experts on valuable data ...

How I Instrumented a Rust Application Without Knowing Rust

As a technical writer, I often have to edit or create code snippets for Splunk's distributions of ...

Splunk Community Platform Survey

Hey Splunk Community, Starting today, the community platform may prompt you to participate in a survey. The ...