Deployment Architecture

Why is only one of my indexers showing a high CPU load percentage?

Contributor

I have six indexers in cluster and 2 search heads. Only one of my indexers is showing >90% CPU load in some other server monitoring tool even when all my inputs are disabled. Also when I run the top command to check the CPU usage of that particular indexer when all my inputs are disabled it shows only 3% to us. What's the best way to check the CPU load and where am I going wrong?

0 Karma

Ultra Champion

We have a similar situation. Looking at the distribution of the daily license across the six indexers helped us a lot. It's on the license master server at licensing.

I assume you would expect an even distribution of the license across the six servers. It's interesting to see what it would show you...

0 Karma

SplunkTrust
SplunkTrust

Here is what I would do to identify the issue.

  • Go to your master and take a look at the DMC, specifically Indexing -> Performance -> Indexing Performance: Instance (select affected indexer)
  • Compare other values from the DMC's Indexing pages with other indexers.
  • Are the indexes configured correct? If you're talking about 6 indexers, I would check the replication settings and also the replication of all indexes. Are they all being replicated?
  • Are you using VMs? If not, check disk health and IOPS, e.g. with Bonnie++
  • In the DMC, you can also take a look at Resource Usage: Instance or Resource Usage: Machine to get some further information.
  • Look at htop/iftop when load is high to get some information which process causes the load which or server is being talked to.

Skalli

0 Karma

SplunkTrust
SplunkTrust

On the server, if you see a low CPU usage then check your monitoring service to make sure that it's reporting correctly.

0 Karma

Contributor

The CPU load is is showing up a high percentage >90% on that indexer. How should I check the CPU load and make it less than 90% like my other five indexers? How to find out what is going wrong with that only indexer?

0 Karma