Deployment Architecture

Why is only one of my indexers showing a high CPU load percentage?

MousumiChowdhur
Contributor

I have six indexers in cluster and 2 search heads. Only one of my indexers is showing >90% CPU load in some other server monitoring tool even when all my inputs are disabled. Also when I run the top command to check the CPU usage of that particular indexer when all my inputs are disabled it shows only 3% to us. What's the best way to check the CPU load and where am I going wrong?

0 Karma

ddrillic
Ultra Champion

We have a similar situation. Looking at the distribution of the daily license across the six indexers helped us a lot. It's on the license master server at licensing.

I assume you would expect an even distribution of the license across the six servers. It's interesting to see what it would show you...

0 Karma

skalliger
SplunkTrust
SplunkTrust

Here is what I would do to identify the issue.

  • Go to your master and take a look at the DMC, specifically Indexing -> Performance -> Indexing Performance: Instance (select affected indexer)
  • Compare other values from the DMC's Indexing pages with other indexers.
  • Are the indexes configured correct? If you're talking about 6 indexers, I would check the replication settings and also the replication of all indexes. Are they all being replicated?
  • Are you using VMs? If not, check disk health and IOPS, e.g. with Bonnie++
  • In the DMC, you can also take a look at Resource Usage: Instance or Resource Usage: Machine to get some further information.
  • Look at htop/iftop when load is high to get some information which process causes the load which or server is being talked to.

Skalli

0 Karma

renjith_nair
SplunkTrust
SplunkTrust

On the server, if you see a low CPU usage then check your monitoring service to make sure that it's reporting correctly.

Happy Splunking!
0 Karma

MousumiChowdhur
Contributor

The CPU load is is showing up a high percentage >90% on that indexer. How should I check the CPU load and make it less than 90% like my other five indexers? How to find out what is going wrong with that only indexer?

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...