Splunk Enterprise Security

What parameter can I modify in limits.conf to solve delayed searches?


What parameter can i modify in limits.conf to solve that?

  • The percentage of non high priority searches delayed (80%) over the last 24 hours is very high and exceeded the red thresholds (20%) on this Splunk instance. Total Searches that were part of this percentage=13378. Total delayed Searches=10799
Labels (1)


You can't just tweak limits.conf to make your schedules work efficiently.

This symptom usually means that you either have too many scheduled searches (reports, alerts, corellation searches) defined altogether or - more probably - you have them squished into the same "schedule spots" as @kkrises suggested. For example - you're trying to run all your searches at 5 minutes past the hour. It's the typical case and you should spread the execution times more evenly throughout the hour, day, or however often your searches run.

In some cases you could try to run more searches at once but that's more tricky and requires more troubleshooting and diagnostics.

Path Finder

@Valen1 - For delayed searches case, I did the below to fix it.

- Monitoring Console in your Search head would help us to determine why searches are delayed. Inside monitoring console, go to Search> Scheduler Activity: Instance. Look for Skip ratio under Runtime statistics dashlet. 

- Identified searches trying to run at the same time.  Reschedule by tweaking the cron schedules of them and skip ratio is reduced.

- Identify searches not completing before the next scheduled run-time, run them in Search app and find the average time taken to complete it. For larger indexes or datamodels especially for network, try to minimize the earliest time range to 1 hour or use summary indexes. 

Hope this helps and an upvote is appreciated. Thank you.


@Valen1 - It's not under limits.conf, instead you can find it under "Set feature indicator threshold" under Health Check of Monitoring Console.

Refer - https://docs.splunk.com/Documentation/Splunk/9.0.0/DMC/Configurefeaturemonitoring


Though I would say this usually indicates, slow Search Head or Indexers in performing searches.


I hope this helps!!!

Get Updates on the Splunk Community!

Introducing the Splunk Community Dashboard Challenge!

Welcome to Splunk Community Dashboard Challenge! This is your chance to showcase your skills in creating ...

Get the T-shirt to Prove You Survived Splunk University Bootcamp

As if Splunk University, in Las Vegas, in-person, with three days of bootcamps and labs weren’t enough, now ...

Wondering How to Build Resiliency in the Cloud?

IT leaders are choosing Splunk Cloud as an ideal cloud transformation platform to drive business resilience,  ...