Monitoring Splunk

How to configure wait time on search heads and indexers?

DarshanBK
Explorer

For one splunk performance issue it was mentioned to us as below

"We have a wait time of 57ms on the Search Head and a wait time of 25ms on the indexer. A wait time of maximum 10ms is good for proper processing in Splunk. This could be the reason for the storage performance issue."

 

Could someone please let us know is there a parameter which denotes wait time on search head and indexers? If yes, please provide us some insight on what is that parameter and details about it.

0 Karma

a_kearney
Path Finder

We had a similar finding from Splunk with high I/O wait time on Search Heads. I have used the folllowing search to monitor

index=_introspection sourcetype=splunk_resource_usage component=IOStats
| eval avg_wait_ms = 'data.avg_total_ms'
| search data.mount_point="/apps/splunk"
| eval sla=10
| timechart limit=30 minspan=60s partial=f avg(data.avg_total_ms) as avg_wait_ms max(sla) AS sla by host

Use a trellis format (split by host) timechart to dispaly. The sla=10 field is to show the 10ms Splunk recommended limit.

I haven't been able to work out why we have high I/O on the Search Heads though, indexer cluster seems to perform OK. The Search Head Captain has notably higher I/O wait compared to others. There has also been issues with KV Store so wondering if that is related.

Note: I/O wait time is not a configuration that can be set. It is the result of the operations being carried out on the disk

0 Karma
Career Survey
First 500 qualified respondents will receive a $20 gift card! Tell us about your professional Splunk journey.
Get Updates on the Splunk Community!

.conf25 Global Broadcast: Don’t Miss a Moment

Hello Splunkers, .conf25 is only a click away.  Not able to make it to .conf25 in person? No worries, you can ...

Observe and Secure All Apps with Splunk

 Join Us for Our Next Tech Talk: Observe and Secure All Apps with SplunkAs organizations continue to innovate ...

What's New in Splunk Observability - August 2025

What's New We are excited to announce the latest enhancements to Splunk Observability Cloud as well as what is ...