Splunk Enterprise Security

What parameter can I modify in limits.conf to solve delayed searches?

Valen1
Engager

What parameter can i modify in limits.conf to solve that?

  • The percentage of non high priority searches delayed (80%) over the last 24 hours is very high and exceeded the red thresholds (20%) on this Splunk instance. Total Searches that were part of this percentage=13378. Total delayed Searches=10799
Labels (1)

PickleRick
SplunkTrust
SplunkTrust

You can't just tweak limits.conf to make your schedules work efficiently.

This symptom usually means that you either have too many scheduled searches (reports, alerts, corellation searches) defined altogether or - more probably - you have them squished into the same "schedule spots" as @kkrises suggested. For example - you're trying to run all your searches at 5 minutes past the hour. It's the typical case and you should spread the execution times more evenly throughout the hour, day, or however often your searches run.

In some cases you could try to run more searches at once but that's more tricky and requires more troubleshooting and diagnostics.

kkrises
Path Finder

@Valen1 - For delayed searches case, I did the below to fix it.

- Monitoring Console in your Search head would help us to determine why searches are delayed. Inside monitoring console, go to Search> Scheduler Activity: Instance. Look for Skip ratio under Runtime statistics dashlet. 

- Identified searches trying to run at the same time.  Reschedule by tweaking the cron schedules of them and skip ratio is reduced.

- Identify searches not completing before the next scheduled run-time, run them in Search app and find the average time taken to complete it. For larger indexes or datamodels especially for network, try to minimize the earliest time range to 1 hour or use summary indexes. 

Hope this helps and an upvote is appreciated. Thank you.

VatsalJagani
SplunkTrust
SplunkTrust

@Valen1 - It's not under limits.conf, instead you can find it under "Set feature indicator threshold" under Health Check of Monitoring Console.

Refer - https://docs.splunk.com/Documentation/Splunk/9.0.0/DMC/Configurefeaturemonitoring

 

Though I would say this usually indicates, slow Search Head or Indexers in performing searches.

 

I hope this helps!!!

Get Updates on the Splunk Community!

Get ready to show some Splunk Certification swagger at .conf24!

Dive into the deep end of data by earning a Splunk Certification at .conf24. We're enticing you again this ...

Built-in Service Level Objectives Management to Bridge the Gap Between Service & ...

Now On-Demand Join us to learn more about how you can leverage Service Level Objectives (SLOs) and the new ...

Database Performance Sidebar Panel Now on APM Database Query Performance & Service ...

We’ve streamlined the troubleshooting experience for database-related service issues by adding a database ...