IOWAIT Mystery - What is it? Is it important?

[eholz1]

Hello All,

The question is is IOWAIT mean anything? I am in the process of upgrading

Splunk 8.2.12 to 9.1.2, and then 9.2.1.  I have not yet upgraded to 9.1.2.

The Health Report is set at default settings i.e. 3, etc.I have tried the suggestion of doubling threshold vales, but eventually get a Warning yellow, or sometimes red, etc.

I am running Splunk Enterprise 8.2.12 on an Oracle Linux (ver 7.9) with 12 cpu and 64GB memory.  Do these settings have any benefit for the IOWAIT thresholds?

I see where I can disable IOWAIT - or does it make any sense to try to generate some sort if Diag, which has a link when opeing the "Health Report Manager"

Any info here? Am I missing something?

Thanks as always for a very helpful Splunk community.

EWHOLZ

I

[eholz1]

thanks, will check

[eholz1]

Hello and Thanks,

It looks like this app, desires to store data in some sort of "cloud" based storage. Is this correct?

The data I have cannot be anywhere but on a private LAN.  I am not sure how to use this app, is there a posting or source of instructions on how to use this SplukBase app?

Thanks for the reply,

ewholz

[deepakc]

Not saying it is, but this could be an indication of CPU being bottlenecked due to Disk IO operations. I might be worth checking your disks are they SDD or align with whats being recommended in the link below.  

See the part on "Notes about optimizing Splunk software and storage usage"  and And "What storage type should I use for a role?"  

https://docs.splunk.com/Documentation/Splunk/9.2.1/Capacity/Referencehardware 

If your disk's are good, then its something else, such as the current set up is being overwhelmed with data and searches, therefore you need to expand, but this all depends on your current design / ingest volumes, and use case.  

The disk's requirements would more or less be the same for the versions of your Splunk.  

Tip: The iostat  command can be used to help with disk iowait (I've had to install this in the past for the TA Nix Add-on and collected that stats that way) .  

[eholz1]

OK, thanks so much, I will review the information you provided. Since our Splunk Enterprise instance is running on a virtual machine - the info you provided could be the issue.

Splunk internals are tricky, I will check things over.

Thanks Again,

EWHOLZ