Splunk Enterprise

Why am I receiving this error message: IOWait - Resource usage?

Path Finder

Hello community, 


I have an issue in my environment and I have been for a while trying to catch the root cause and I feel I am not even close.

I am receiving this message frequently:


And I don't know where this come from:

I checked the %iowait at the SO and never is up to 0.02 but the alert about IOWait is stilling coming for search heads and indexers as well.



I checked the resources and there is not issue:


Also I check the CPU running this search and by the MC and there is not a huge use of the CPU.

This is for the last 4 hours


So I am really confused, I don't know if I missing something.

Version is 8.2.2 - Cluster environment.

Can you please can help me on this?

Kind Regards.



If you are running this on VMware then couple of things what you should check/fix:

  • Don't use too many vCPU vs core count on individual socket on host. If VM uses cores from more than one socket it affects performance! Much better to use enough low amount of cores than spread those to to socket
  • Never ever over allocate mem or cpu on those host where you are running Splunk VMs!
  • Have you enough IOPS on host level? Basically it should have min. 800 IOPS * amount of Splunk nodes on that VMFS + something for other VMs too

r. Ismo

0 Karma

Path Finder

Thank for the answer, just one question:

Can you please explain this point a little bit more:

  • Never ever over allocate mem or cpu on those host where you are running Splunk VMs!

Why not?


Thank you


0 Karma


In technical point of view it's quite expensive operation to move especially memory to one VM to another. This has huge performance effects for Splunk VMs and it's not a recommended configuration for Splunk VMs.

Here is couple of old answers related to this. If I recall right there is also some White paper or other technical documentation about running Splunk on VMware, but I cannot found those now.

0 Karma
Get Updates on the Splunk Community!

Data Preparation Made Easy: SPL2 for Edge Processor

By now, you may have heard the exciting news that Edge Processor, the easy-to-use Splunk data preparation tool ...

Introducing Edge Processor: Next Gen Data Transformation

We get it - not only can it take a lot of time, money and resources to get data into Splunk, but it also takes ...

Tips & Tricks When Using Ingest Actions

Tune in to learn about:Large scale architecture when using Ingest ActionsRegEx performance considerations ...