Splunk Enterprise

Why am I receiving this error message: IOWait - Resource usage?

Path Finder

Hello community, 


I have an issue in my environment and I have been for a while trying to catch the root cause and I feel I am not even close.

I am receiving this message frequently:


And I don't know where this come from:

I checked the %iowait at the SO and never is up to 0.02 but the alert about IOWait is stilling coming for search heads and indexers as well.



I checked the resources and there is not issue:


Also I check the CPU running this search and by the MC and there is not a huge use of the CPU.

This is for the last 4 hours


So I am really confused, I don't know if I missing something.

Version is 8.2.2 - Cluster environment.

Can you please can help me on this?

Kind Regards.



If you are running this on VMware then couple of things what you should check/fix:

  • Don't use too many vCPU vs core count on individual socket on host. If VM uses cores from more than one socket it affects performance! Much better to use enough low amount of cores than spread those to to socket
  • Never ever over allocate mem or cpu on those host where you are running Splunk VMs!
  • Have you enough IOPS on host level? Basically it should have min. 800 IOPS * amount of Splunk nodes on that VMFS + something for other VMs too

r. Ismo

0 Karma

Path Finder

Thank for the answer, just one question:

Can you please explain this point a little bit more:

  • Never ever over allocate mem or cpu on those host where you are running Splunk VMs!

Why not?


Thank you


0 Karma


In technical point of view it's quite expensive operation to move especially memory to one VM to another. This has huge performance effects for Splunk VMs and it's not a recommended configuration for Splunk VMs.

Here is couple of old answers related to this. If I recall right there is also some White paper or other technical documentation about running Splunk on VMware, but I cannot found those now.

0 Karma
Get Updates on the Splunk Community!

Don't wait! Accept the Mission Possible: Splunk Adoption Challenge Now and Win ...

Attention everyone! We have exciting news to share! We are recruiting new members for the Mission Possible: ...

Unify Your SecOps with Splunk Mission Control

In today’s post, I'm excited to share some recent Splunk Mission Control innovations. With Splunk Mission ...

Data Preparation Made Easy: SPL2 for Edge Processor

By now, you may have heard the exciting news that Edge Processor, the easy-to-use Splunk data preparation tool ...