I received red alarms from health status.
The types of alarm vary over time.
but the warnings that continuously occur are Ingestion Latency, IOWait, Searches Delayed, etc.
And the Detail message displays 'Splunkd's processing queue is full.'
Is there any way to check which process is in the queue?
OR is there a way to flush the queue?
I increased CPU and memory, but the problem was not solved.
And I recently upgraded the Splunk version from 8.1.4 to 9.0.2.
Thank you.
Ingestion latency might be a "feature" of your setup in certain cases (like big periodically created - for example, rsynced from remote location - files and low thruput limit).
IOwait is, as @richgalloway said, a known false positive.
Searches delayed is typically an issue with your workload and/or definition of your searching schedules.
Apart from IOwait they should not be connected with your upgrade.
The IOWait warning is a known false positive. You can adjust it to turn red with a higher IOWait value.
The Monitoring Console will display the indexer queues at Indexing->Performance->Indexing Performance: Instance