topic Re: CPU consumption is very high in Splunk indexers in Splunk Enterprise

CPU consumption is very high in Splunk indexers

shadysplunker — Mon, 27 Nov 2023 10:28:52 GMT

We have 24 indexers in an indexer cluster. Recently the CPU usage is almost 100%, not on all the indexers but it fluctuates between the indexers. Under indexer clustering section, I can see the status going to "Pending" randomly between the indexers for few seconds. It is very continuous and also causes an increase in the number of fixup buckets.

I have restarted the indexer servers manually where I saw high CPU load, but it did not resolve the issue. What would be the best option to fix this and the possible root cause?

Any suggestions would be very helpful.

Thanks in advance!

Re: CPU consumption is very high in Splunk indexers

isoutamo — Mon, 27 Nov 2023 13:00:40 GMT

There could be several reasons why indexers have high CPU%. It's really hard to make correct guess without seeing what there is happening via MC. I'm suspecting that you have MC on place? If you have, then use it and if You haven't then it's time to setup it now.

Under MC there are several places where you could look in which part that issue could be:

ingesting
- which pipeline
Disk I/O etc.
searching
- indexer vs. sh side

Also there must be good understanding which kind of environment you have to make guesses where that issue could be.

Best option will be if you could as some splunk partner/specialist or Splunk Professional services to look your environment.

r. Ismo

Re: CPU consumption is very high in Splunk indexers

shadysplunker — Tue, 28 Nov 2023 07:28:10 GMT

Hi,

I noticed that ingestion latency health check is turning yellow for indexers. Even there is a delay in searching for the data. The index queues shows blocked while checking the internal logs of heavy forwarders.

Could performing a rolling restart of indexers help since it has been a very long time now doing that.

Re: CPU consumption is very high in Splunk indexers

PickleRick — Tue, 28 Nov 2023 11:29:07 GMT

Without any concrete data it's just fortune telling.

Check processes, check i/o saturation, check memory usage. Verify if it's even Splunk that's causing cpu hogging.

Restarting processes blindly will not help much probably without addressing the underlying cause.

Has anything been changed recently? Upgraded?