Splunk Enterprise

Why am I receiving this error message: IOWait - Resource usage?

Path Finder

Hello community, 


I have an issue in my environment and I have been for a while trying to catch the root cause and I feel I am not even close.

I am receiving this message frequently:


And I don't know where this come from:

I checked the %iowait at the SO and never is up to 0.02 but the alert about IOWait is stilling coming for search heads and indexers as well.



I checked the resources and there is not issue:


Also I check the CPU running this search and by the MC and there is not a huge use of the CPU.

This is for the last 4 hours


So I am really confused, I don't know if I missing something.

Version is 8.2.2 - Cluster environment.

Can you please can help me on this?

Kind Regards.



If you are running this on VMware then couple of things what you should check/fix:

  • Don't use too many vCPU vs core count on individual socket on host. If VM uses cores from more than one socket it affects performance! Much better to use enough low amount of cores than spread those to to socket
  • Never ever over allocate mem or cpu on those host where you are running Splunk VMs!
  • Have you enough IOPS on host level? Basically it should have min. 800 IOPS * amount of Splunk nodes on that VMFS + something for other VMs too

r. Ismo

0 Karma

Path Finder

Thank for the answer, just one question:

Can you please explain this point a little bit more:

  • Never ever over allocate mem or cpu on those host where you are running Splunk VMs!

Why not?


Thank you


0 Karma


In technical point of view it's quite expensive operation to move especially memory to one VM to another. This has huge performance effects for Splunk VMs and it's not a recommended configuration for Splunk VMs.

Here is couple of old answers related to this. If I recall right there is also some White paper or other technical documentation about running Splunk on VMware, but I cannot found those now.

0 Karma
Get Updates on the Splunk Community!

Optimize Cloud Monitoring

  TECH TALKS Optimize Cloud Monitoring Tuesday, August 13, 2024  |  11:00AM–12:00PM PST   Register to ...

What's New in Splunk Cloud Platform 9.2.2403?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.2.2403! Analysts can ...

Stay Connected: Your Guide to July and August Tech Talks, Office Hours, and Webinars!

Dive into our sizzling summer lineup for July and August Community Office Hours and Tech Talks. Scroll down to ...