Splunk Enterprise

Why is the health status of IOWait red?

aateeq
Explorer

After I successfully installed Splunk Enterprise and I'd added some data I tried to do some searching, but there was an issue with searching, and the intended result doesn't appear at all. I think this is due to the red health status for IOWait as shown below, so how could solve this issue?45.png

Labels (2)
Tags (1)
0 Karma

_joe
Communicator

Go to Settings > Health report manager and edit the threshold for IOWait. Double the thresholds you are currently hitting. 

https://docs.splunk.com/Documentation/Splunk/8.2.3/DMC/Configurefeaturemonitoring#Disable_a_feature

0 Karma

SinghK
Builder

i have seen this in other environments too.. let me see if i can get the attention to this, seems like a never ending issue.

0 Karma

_joe
Communicator

After using the health reporter analyzer for a few weeks, I will agree, it really doesn't seem to be very accurate in a distributed/clustered environment.

I have massively overpowered environments that are still getting IOWait alerts even after raising the thresholds. 

I've also seen suggestions to raise the suppression status. I am trying it.

suppress_status_update_ms = 30000
* Default: 300.

https://docs.splunk.com/Documentation/Splunk/8.2.4/Admin/Healthconf

0 Karma

shivanshu1593
Builder

Hello @aateeq ,

This is a disk performance issue. Looks like the IO of the disk, where you've installed Splunk is very low. You'll have to check and verify the throuput of your disks, Splunk requires at least 800 iops. The more the better. You can use tools like bonnie++ to measure the IO of a disk.

Also, please make sure that there's enough space on the partition, where Splunk is installed.

Hope this helps.

Thanks,

S

***If this helped, please accept it as a solution. It helps others to find the solution for similar issues quickly.***

Thank you,
Shiv
###If you found the answer helpful, kindly consider upvoting/accepting it as the answer as it helps other Splunkers find the solutions to similar issues###
0 Karma

vzabawski
Path Finder

I'm using Azure Premium SSD which should have max IOPS of 20,000, according to the documentation, but I'll run a test to see some real-life results.

0 Karma

SinghK
Builder

Have you by any chance upgraded to 8.2.5 if yes then IOPS values are a bit sensitive. 

0 Karma

vzabawski
Path Finder

Yes, I've upgraded from 7.x to 8.2.x and after that I've started receiving those notifications.

I've measured iops and seems like everything is fine:

fiotest: (groupid=0, jobs=1): err= 0: pid=29: Tue May 24 13:40:16 2022
read: IOPS=1796, BW=7186KiB/s (7358kB/s)(6141MiB/875157msec)
bw ( KiB/s): min= 5584, max= 9976, per=100.00%, avg=7192.71, stdev=565.38, samples=1748
iops : min= 1396, max= 2494, avg=1798.08, stdev=141.33, samples=1748
write: IOPS=599, BW=2400KiB/s (2457kB/s)(2051MiB/875157msec); 0 zone resets
bw ( KiB/s): min= 1888, max= 2885, per=100.00%, avg=2401.51, stdev=73.89, samples=1748
iops : min= 472, max= 721, avg=600.34, stdev=18.47, samples=1748
cpu : usr=0.89%, sys=2.59%, ctx=533118, majf=0, minf=6
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued rwts: total=1572145,525007,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=64
0 Karma

SinghK
Builder

What version of splunk are you on?

0 Karma

vzabawski
Path Finder

Splunk 8.2.5, but I'm getting this alert starting from Splunk 8.1.6 (if I'm not mistaken).

0 Karma

SinghK
Builder

I have reported this as a bug lets see what they say.

0 Karma

shivanshu1593
Builder

I'd suggest checking the actual IOPS using dd or Bonnie ++. The issue almost certainly is due to the low IOPS. Why is it happening can be checked by these tools.

https://www.jamescoyle.net/how-to/599-benchmark-disk-io-with-dd-and-bonnie

Thank you,
Shiv
###If you found the answer helpful, kindly consider upvoting/accepting it as the answer as it helps other Splunkers find the solutions to similar issues###
0 Karma

SinghK
Builder

Give me few mins I am checking how can we escalate it .

0 Karma

aateeq
Explorer

@shivanshu1593  Thanks for your reply.

I installed the Linux machine in a Vbox with 40G disk space, so I think I have enough disk space for Splunk. But, regarding the throughput of the disk how can I check it and increase it to more than 800 iops؟

0 Karma

shivanshu1593
Builder

You can go through the following link and use either dd or bonnie++ to check the IOPS. 

https://www.jamescoyle.net/how-to/599-benchmark-disk-io-with-dd-and-bonnie

 

Thanks,

S

Thank you,
Shiv
###If you found the answer helpful, kindly consider upvoting/accepting it as the answer as it helps other Splunkers find the solutions to similar issues###
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...