Why is the health status of IOWait red?

[aateeq]

After I successfully installed Splunk Enterprise and I'd added some data I tried to do some searching, but there was an issue with searching, and the intended result doesn't appear at all. I think this is due to the red health status for IOWait as shown below, so how could solve this issue?

[_joe]

Go to Settings > Health report manager and edit the threshold for IOWait. Double the thresholds you are currently hitting. 

https://docs.splunk.com/Documentation/Splunk/8.2.3/DMC/Configurefeaturemonitoring#Disable_a_feature

[SinghK]

i have seen this in other environments too.. let me see if i can get the attention to this, seems like a never ending issue.

[_joe]

After using the health reporter analyzer for a few weeks, I will agree, it really doesn't seem to be very accurate in a distributed/clustered environment.

I have massively overpowered environments that are still getting IOWait alerts even after raising the thresholds. 

I've also seen suggestions to raise the suppression status. I am trying it.

suppress_status_update_ms = 30000
* Default: 300.

https://docs.splunk.com/Documentation/Splunk/8.2.4/Admin/Healthconf

[shivanshu1593]

Hello @aateeq ,

This is a disk performance issue. Looks like the IO of the disk, where you've installed Splunk is very low. You'll have to check and verify the throuput of your disks, Splunk requires at least 800 iops. The more the better. You can use tools like bonnie++ to measure the IO of a disk.

Also, please make sure that there's enough space on the partition, where Splunk is installed.

Hope this helps.

Thanks,

S

***If this helped, please accept it as a solution. It helps others to find the solution for similar issues quickly.***

[vzabawski]

I'm using Azure Premium SSD which should have max IOPS of 20,000, according to the documentation, but I'll run a test to see some real-life results.

[SinghK]

Have you by any chance upgraded to 8.2.5 if yes then IOPS values are a bit sensitive. 

[vzabawski]

Yes, I've upgraded from 7.x to 8.2.x and after that I've started receiving those notifications.

I've measured iops and seems like everything is fine:

fiotest: (groupid=0, jobs=1): err= 0: pid=29: Tue May 24 13:40:16 2022
read: IOPS=1796, BW=7186KiB/s (7358kB/s)(6141MiB/875157msec)
bw ( KiB/s): min= 5584, max= 9976, per=100.00%, avg=7192.71, stdev=565.38, samples=1748
iops : min= 1396, max= 2494, avg=1798.08, stdev=141.33, samples=1748
write: IOPS=599, BW=2400KiB/s (2457kB/s)(2051MiB/875157msec); 0 zone resets
bw ( KiB/s): min= 1888, max= 2885, per=100.00%, avg=2401.51, stdev=73.89, samples=1748
iops : min= 472, max= 721, avg=600.34, stdev=18.47, samples=1748
cpu : usr=0.89%, sys=2.59%, ctx=533118, majf=0, minf=6
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued rwts: total=1572145,525007,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=64

[SinghK]

What version of splunk are you on?

[vzabawski]

Splunk 8.2.5, but I'm getting this alert starting from Splunk 8.1.6 (if I'm not mistaken).

[SinghK]

I have reported this as a bug lets see what they say.

[shivanshu1593]

I'd suggest checking the actual IOPS using dd or Bonnie ++. The issue almost certainly is due to the low IOPS. Why is it happening can be checked by these tools.

https://www.jamescoyle.net/how-to/599-benchmark-disk-io-with-dd-and-bonnie

[SinghK]

Give me few mins I am checking how can we escalate it .

[aateeq]

@shivanshu1593  Thanks for your reply.

I installed the Linux machine in a Vbox with 40G disk space, so I think I have enough disk space for Splunk. But, regarding the throughput of the disk how can I check it and increase it to more than 800 iops؟

[shivanshu1593]

You can go through the following link and use either dd or bonnie++ to check the IOPS. 

https://www.jamescoyle.net/how-to/599-benchmark-disk-io-with-dd-and-bonnie

Thanks,

S