Splunk Enterprise

Why am I receiving this error message: IOWait - Resource usage?

glpadilla_sol
Path Finder

Hello community, 

 

I have an issue in my environment and I have been for a while trying to catch the root cause and I feel I am not even close.

I am receiving this message frequently:

glpadilla_sol_0-1639168677685.png

And I don't know where this come from:

I checked the %iowait at the SO and never is up to 0.02 but the alert about IOWait is stilling coming for search heads and indexers as well.

glpadilla_sol_1-1639168718833.png

 

I checked the resources and there is not issue:

glpadilla_sol_2-1639168824049.png

Also I check the CPU running this search and by the MC and there is not a huge use of the CPU.

This is for the last 4 hours

glpadilla_sol_3-1639168899835.png

So I am really confused, I don't know if I missing something.

Version is 8.2.2 - Cluster environment.

Can you please can help me on this?

Kind Regards.

isoutamo
SplunkTrust
SplunkTrust

Hi

If you are running this on VMware then couple of things what you should check/fix:

  • Don't use too many vCPU vs core count on individual socket on host. If VM uses cores from more than one socket it affects performance! Much better to use enough low amount of cores than spread those to to socket
  • Never ever over allocate mem or cpu on those host where you are running Splunk VMs!
  • Have you enough IOPS on host level? Basically it should have min. 800 IOPS * amount of Splunk nodes on that VMFS + something for other VMs too

r. Ismo

0 Karma

glpadilla_sol
Path Finder

Thank for the answer, just one question:

Can you please explain this point a little bit more:

  • Never ever over allocate mem or cpu on those host where you are running Splunk VMs!

Why not?

 

Thank you

 

0 Karma

isoutamo
SplunkTrust
SplunkTrust

In technical point of view it's quite expensive operation to move especially memory to one VM to another. This has huge performance effects for Splunk VMs and it's not a recommended configuration for Splunk VMs.

Here is couple of old answers related to this. If I recall right there is also some White paper or other technical documentation about running Splunk on VMware, but I cannot found those now.

0 Karma
Get Updates on the Splunk Community!

Splunk Training for All: Meet Aspiring Cybersecurity Analyst, Marc Alicea

Splunk Education believes in the value of training and certification in today’s rapidly-changing data-driven ...

Investigate Security and Threat Detection with VirusTotal and Splunk Integration

As security threats and their complexities surge, security analysts deal with increased challenges and ...

Observability Highlights | January 2023 Newsletter

 January 2023New Product Releases Splunk Network Explorer for Infrastructure MonitoringSplunk unveils Network ...