Found in a post. Please follow the below troubleshooting steps. A large amount of dirs/files can slow things down. $SPLUNK_HOME/var/run/splunk/dispatch or if in a pooled space [Pooled Share]/var/run/splunk/dispatch to get a count of files/dirs in each directory ls -l|wc -l You might want to check for a large amount of files under the var dirs in general. Here's also a search to calculate scheduled search lag to see if the scheduler is lagging. 30 seconds lag is probably normal but you may want to investigate above that. you can set the HIGH_WATERMARK to your liking as a reference point. As a requirement, you will need to be indexing the scheduler.log replace host names below with host names for your search heads (host=hosta OR host=hostb) index=_internal source=*scheduler.log |eval JOB_DELAY_SECS=(dispatch_time-scheduled_time)|timechart span=5m perc95(JOB_DELAY_SECS) by host|eval HIGH_WATERMARK=100 If you are on Linux, you can run this command to see what splunkd or splunkweb is spending time on. strace -p <splunk pid> -tt
... View more
