I have an issue in my environment and I have been for a while trying to catch the root cause and I feel I am not even close.
I am receiving this message frequently:
And I don't know where this come from:
I checked the %iowait at the SO and never is up to 0.02 but the alert about IOWait is stilling coming for search heads and indexers as well.
I checked the resources and there is not issue:
Also I check the CPU running this search and by the MC and there is not a huge use of the CPU.
This is for the last 4 hours
So I am really confused, I don't know if I missing something.
Version is 8.2.2 - Cluster environment.
Can you please can help me on this?
If you are running this on VMware then couple of things what you should check/fix:
Thank for the answer, just one question:
Can you please explain this point a little bit more:
In technical point of view it's quite expensive operation to move especially memory to one VM to another. This has huge performance effects for Splunk VMs and it's not a recommended configuration for Splunk VMs.
Here is couple of old answers related to this. If I recall right there is also some White paper or other technical documentation about running Splunk on VMware, but I cannot found those now.