Deployment Architecture

Why is only one of my indexers showing a high CPU load percentage?

MousumiChowdhur
Contributor

I have six indexers in cluster and 2 search heads. Only one of my indexers is showing >90% CPU load in some other server monitoring tool even when all my inputs are disabled. Also when I run the top command to check the CPU usage of that particular indexer when all my inputs are disabled it shows only 3% to us. What's the best way to check the CPU load and where am I going wrong?

0 Karma

ddrillic
Ultra Champion

We have a similar situation. Looking at the distribution of the daily license across the six indexers helped us a lot. It's on the license master server at licensing.

I assume you would expect an even distribution of the license across the six servers. It's interesting to see what it would show you...

0 Karma

skalliger
Motivator

Here is what I would do to identify the issue.

  • Go to your master and take a look at the DMC, specifically Indexing -> Performance -> Indexing Performance: Instance (select affected indexer)
  • Compare other values from the DMC's Indexing pages with other indexers.
  • Are the indexes configured correct? If you're talking about 6 indexers, I would check the replication settings and also the replication of all indexes. Are they all being replicated?
  • Are you using VMs? If not, check disk health and IOPS, e.g. with Bonnie++
  • In the DMC, you can also take a look at Resource Usage: Instance or Resource Usage: Machine to get some further information.
  • Look at htop/iftop when load is high to get some information which process causes the load which or server is being talked to.

Skalli

0 Karma

renjith_nair
Legend

On the server, if you see a low CPU usage then check your monitoring service to make sure that it's reporting correctly.

---
What goes around comes around. If it helps, hit it with Karma 🙂
0 Karma

MousumiChowdhur
Contributor

The CPU load is is showing up a high percentage >90% on that indexer. How should I check the CPU load and make it less than 90% like my other five indexers? How to find out what is going wrong with that only indexer?

0 Karma
Get Updates on the Splunk Community!

Demo Day: Strengthen Your SOC with Splunk Enterprise Security 8.1

Today’s threat landscape is more complex than ever. Security operation centers (SOCs) are overwhelmed with ...

Dashboards: Hiding charts while search is being executed and other uses for tokens

There are a couple of features of SimpleXML / Classic dashboards that can be used to enhance the user ...

Splunk Observability Cloud's AI Assistant in Action Series: Explaining Metrics and ...

This is the fourth post in the Splunk Observability Cloud’s AI Assistant in Action series that digs into how ...