topic Re: Health status of IOWait is read!! in Splunk Enterprise

Why is the health status of IOWait red?

aateeq — Thu, 10 Feb 2022 00:50:46 GMT

After I successfully installed Splunk Enterprise and I'd added some data I tried to do some searching, but there was an issue with searching, and the intended result doesn't appear at all. I think this is due to the red health status for IOWait as shown below, so how could solve this issue?

Re: Health status of IOWait is read!!

shivanshu1593 — Sat, 04 Sep 2021 12:12:37 GMT

Hello @aateeq ,

This is a disk performance issue. Looks like the IO of the disk, where you've installed Splunk is very low. You'll have to check and verify the throuput of your disks, Splunk requires at least 800 iops. The more the better. You can use tools like bonnie++ to measure the IO of a disk.

Also, please make sure that there's enough space on the partition, where Splunk is installed.

Hope this helps.

Thanks,

***If this helped, please accept it as a solution. It helps others to find the solution for similar issues quickly.***

Re: Health status of IOWait is read!!

aateeq — Sat, 04 Sep 2021 12:44:49 GMT

@shivanshu1593 Thanks for your reply.

I installed the Linux machine in a Vbox with 40G disk space, so I think I have enough disk space for Splunk. But, regarding the throughput of the disk how can I check it and increase it to more than 800 iops؟

Re: Health status of IOWait is read!!

shivanshu1593 — Sat, 04 Sep 2021 13:08:51 GMT

You can go through the following link and use either dd or bonnie++ to check the IOPS.

https://www.jamescoyle.net/how-to/599-benchmark-disk-io-with-dd-and-bonnie

Thanks,

Re: Health status of IOWait is read!!

_joe — Thu, 13 Jan 2022 12:32:45 GMT

Go to Settings > Health report manager and edit the threshold for IOWait. Double the thresholds you are currently hitting.

https://docs.splunk.com/Documentation/Splunk/8.2.3/DMC/Configurefeaturemonitoring#Disable_a_feature

Re: Health status of IOWait is read!!

_joe — Wed, 09 Feb 2022 13:50:42 GMT

After using the health reporter analyzer for a few weeks, I will agree, it really doesn't seem to be very accurate in a distributed/clustered environment.

I have massively overpowered environments that are still getting IOWait alerts even after raising the thresholds.

I've also seen suggestions to raise the suppression status. I am trying it.

suppress_status_update_ms = 30000
* Default: 300.

https://docs.splunk.com/Documentation/Splunk/8.2.4/Admin/Healthconf

Re: Health status of IOWait is read!!

vzabawski — Tue, 24 May 2022 12:35:37 GMT

I'm using Azure Premium SSD which should have max IOPS of 20,000, according to the documentation, but I'll run a test to see some real-life results.

Re: Health status of IOWait is read!!

SinghK — Tue, 24 May 2022 12:39:05 GMT

Have you by any chance upgraded to 8.2.5 if yes then IOPS values are a bit sensitive.

Re: Health status of IOWait is read!!

vzabawski — Tue, 24 May 2022 14:35:07 GMT

Yes, I've upgraded from 7.x to 8.2.x and after that I've started receiving those notifications.

I've measured iops and seems like everything is fine:

fiotest: (groupid=0, jobs=1): err= 0: pid=29: Tue May 24 13:40:16 2022 read: IOPS=1796, BW=7186KiB/s (7358kB/s)(6141MiB/875157msec) bw ( KiB/s): min= 5584, max= 9976, per=100.00%, avg=7192.71, stdev=565.38, samples=1748 iops : min= 1396, max= 2494, avg=1798.08, stdev=141.33, samples=1748 write: IOPS=599, BW=2400KiB/s (2457kB/s)(2051MiB/875157msec); 0 zone resets bw ( KiB/s): min= 1888, max= 2885, per=100.00%, avg=2401.51, stdev=73.89, samples=1748 iops : min= 472, max= 721, avg=600.34, stdev=18.47, samples=1748 cpu : usr=0.89%, sys=2.59%, ctx=533118, majf=0, minf=6 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0% issued rwts: total=1572145,525007,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=64

Re: Health status of IOWait is read!!

SinghK — Tue, 24 May 2022 14:42:26 GMT

What version of splunk are you on?

Re: Health status of IOWait is read!!

SinghK — Tue, 24 May 2022 14:49:29 GMT

i have seen this in other environments too.. let me see if i can get the attention to this, seems like a never ending issue.

Re: Health status of IOWait is read!!

vzabawski — Tue, 24 May 2022 14:52:55 GMT

Splunk 8.2.5, but I'm getting this alert starting from Splunk 8.1.6 (if I'm not mistaken).

Re: Health status of IOWait is read!!

SinghK — Tue, 24 May 2022 14:54:35 GMT

Give me few mins I am checking how can we escalate it .

Re: Health status of IOWait is read!!

shivanshu1593 — Tue, 24 May 2022 14:55:33 GMT

I'd suggest checking the actual IOPS using dd or Bonnie ++. The issue almost certainly is due to the low IOPS. Why is it happening can be checked by these tools.

https://www.jamescoyle.net/how-to/599-benchmark-disk-io-with-dd-and-bonnie

Re: Health status of IOWait is read!!

SinghK — Tue, 24 May 2022 15:05:23 GMT

I have reported this as a bug lets see what they say.