Splunk Search

Most of searches are getting deferred

Rukmani_Splunk
Path Finder

Hi all,
We are using SoS app for monitoring our schedules. We are working on reducing the schedules which are being skipped . But what is case about the deferred searches ? Its keep on increasing ? how to reduce them.
thoughts pls

Tags (1)

yannK
Splunk Employee
Splunk Employee

A deferred search is a search that couldn't be executed right now, because of the system or role search concurrency limit. Therefore they are executed a few seconds later. This is an expected behavior.
By example is you have a dashboard with 10 searches, but a limit of 6 concurrent searches, some panels will load after the first ones completed.

If you look at the audit logs, you can find how long they were deferred before being executed.
If a search is deferred too long, it will finally be skipped: skipped searches

The root cause are usually caused by :
- too many searches : you have too many searches (or heavy dashboard)
- non optimized scheduled searches taking long to run and overlapping.
- hardware limit : the indexers and search-head have not enough cpu core to handle high search concurrency. (check limits.conf), on 6.0 the formula for historical search concurrency = 6+ 1* (number of cores)

(hint, disable the deployment monitor app if you have it)

Rukmani_Splunk
Path Finder

Thanks a lot

0 Karma

MuS
Legend

more an advice then an answer, check out this answer on search scheduling http://answers.splunk.com/answers/33717/scheduled-searches-for-summary-index-does-not-run-no-skipped...

Get Updates on the Splunk Community!

Updated Team Landing Page in Splunk Observability

We’re making some changes to the team landing page in Splunk Observability, based on your feedback. The ...

New! Splunk Observability Search Enhancements for Splunk APM Services/Traces and ...

Regardless of where you are in Splunk Observability, you can search for relevant APM targets including service ...

Webinar Recap | Revolutionizing IT Operations: The Transformative Power of AI and ML ...

The Transformative Power of AI and ML in Enhancing Observability   In the realm of IT operations, the ...