topic Re: High CPU Usage on Splunk Indexers in Getting Data In

High CPU Usage on Splunk Indexers

alexspunkshell — Wed, 29 Jul 2020 15:07:10 GMT

Frequently i am receiving high CPU Usage alerts with over 99% on all 3 indexers.

I am unable to search any query. It shows waiting for quequed job to start.

Please help me here. How to check the issue and resolve it.

Re: High CPU Usage on Splunk Indexers

richgalloway — Wed, 29 Jul 2020 17:37:13 GMT

Check the Monitoring Console (Settings->Monitoring Console->Indexing->Performance->Indexing Performance: Advanced) for insights.

Re: High CPU Usage on Splunk Indexers

alexspunkshell — Tue, 04 Aug 2020 13:16:25 GMT

@richgalloway Thanks for your reply.

Checked the Monitoring Console (Settings->Monitoring Console->Indexing->Performance->Indexing Performance: Advanced)

But its not showing any details.

Showing no results found. Any other alternative ways to get a solution?

Re: High CPU Usage on Splunk Indexers

richgalloway — Tue, 04 Aug 2020 14:08:43 GMT

"No results found" means the MC is not configured. Make sure you're signed in to the right Splunk instance and that the MC is set up.

Re: High CPU Usage on Splunk Indexers

shivanshu1593 — Thu, 06 Aug 2020 10:48:07 GMT

Hello,

In order to find the root cause, in your Indexers, go to Monitoring console -> Resource usage -> Resource usage: Instance.

Under the snapshots section, you'll find two very important graphs. Physical memory usage by process class and CPU Usage by process class.

You'd want to look at the CPU usage graphs and see what's causing the hogging of CPU utilisation. If it says Search, that means you have to look at people running resource intensive searches in your environment. For that, you'd want to go to Search -> Search Activity: Instance and check everything out from there.

You can always create alerts off the audit data to find violators, who are running long running searches, or too many searches etc. If it's not a search issue, please contact Splunk support, as it maybe a case of memory leakage.

Here's the article that you can also go through for this:

Resouce Usage CPU - Splunk Docs

Hope this helps,

Note: If this helped, please mark it as an accepted answer.

Re: High CPU Usage on Splunk Indexers

bakkre — Wed, 28 Jul 2021 12:17:55 GMT

We also had high CPU usage on our indexers in a test enviroment and query's took a very long (45 min)time.

In the monitoring of the servers we noticed that the cpu ready time was between 1 and 5 Ms. The usage of the server was 100%

The server had 8 Vcpu's.

By reducing the number of vcpu's to 2 vcpu;s per server the cpu ready time reduced.

This gave a large peformance boost on the indexers query's time was reduced for the same query to 5 Min

So my advice if you are running in a virtualized enviroment play with the number of vcpu's to get the optimal peformance. I know the say minimum 16 vCpu but if you have high ready times it is worth to try.

Look at the number of MHZ the server Uses and divide this by the speed of your cores en set this number of vcpu's to begin with.

Re: High CPU Usage on Splunk Indexers

gcusello — Wed, 28 Jul 2021 12:33:47 GMT

Hi @alexspunkshell,

At first, some quick questions:

what's your hardware configuration (CPUs and RAM)?
did you used the recommended hardware references? https://docs.splunk.com/Documentation/Splunk/8.2.1/Capacity/Referencehardware
which apps are you using? Enterprise Security, Security Essentials, etc...
what storage are you using? (Splunk recommends at least 800 IOPS).

If you're using a correct HW configuration and you haven't special requirements from some App, you can see if there are some heavy scheduled searches that make your system busy.

E.g., if you're using Real Time Searches, you tale a CPU for each search you're sunning, so if you have some real time search with one or two subsearches you're filling your system.

Then, are you usung searches with transaction or join commands? they are very expensive for resources.

You can check the running searches, as @richgalloway said, using the Monitoring Console [Settings -- Resource Usage -- CPU Usage: Instance].

Ciao.

Giuseppe