topic Why am I receiving this error message: IOWait - Resource usage? in Splunk Enterprise

Why am I receiving this error message: IOWait - Resource usage?

glpadilla_sol — Tue, 28 Jun 2022 17:19:28 GMT

Hello community,

I have an issue in my environment and I have been for a while trying to catch the root cause and I feel I am not even close.

I am receiving this message frequently:

And I don't know where this come from:

I checked the %iowait at the SO and never is up to 0.02 but the alert about IOWait is stilling coming for search heads and indexers as well.

I checked the resources and there is not issue:

Also I check the CPU running this search and by the MC and there is not a huge use of the CPU.

This is for the last 4 hours

So I am really confused, I don't know if I missing something.

Version is 8.2.2 - Cluster environment.

Can you please can help me on this?

Kind Regards.

Re: IOWait - Resource usage

isoutamo — Tue, 28 Jun 2022 16:18:49 GMT

If you are running this on VMware then couple of things what you should check/fix:

Don't use too many vCPU vs core count on individual socket on host. If VM uses cores from more than one socket it affects performance! Much better to use enough low amount of cores than spread those to to socket
Never ever over allocate mem or cpu on those host where you are running Splunk VMs!
Have you enough IOPS on host level? Basically it should have min. 800 IOPS * amount of Splunk nodes on that VMFS + something for other VMs too

r. Ismo

Re: IOWait - Resource usage

glpadilla_sol — Tue, 28 Jun 2022 17:16:31 GMT

Thank for the answer, just one question:

Can you please explain this point a little bit more:

Never ever over allocate mem or cpu on those host where you are running Splunk VMs!

Why not?

Thank you

Re: IOWait - Resource usage

isoutamo — Tue, 28 Jun 2022 17:26:54 GMT

In technical point of view it's quite expensive operation to move especially memory to one VM to another. This has huge performance effects for Splunk VMs and it's not a recommended configuration for Splunk VMs.

Here is couple of old answers related to this. If I recall right there is also some White paper or other technical documentation about running Splunk on VMware, but I cannot found those now.