Splunk Search

Most of searches are getting deferred

Rukmani_Splunk
Path Finder

Hi all,
We are using SoS app for monitoring our schedules. We are working on reducing the schedules which are being skipped . But what is case about the deferred searches ? Its keep on increasing ? how to reduce them.
thoughts pls

Tags (1)

yannK
Splunk Employee
Splunk Employee

A deferred search is a search that couldn't be executed right now, because of the system or role search concurrency limit. Therefore they are executed a few seconds later. This is an expected behavior.
By example is you have a dashboard with 10 searches, but a limit of 6 concurrent searches, some panels will load after the first ones completed.

If you look at the audit logs, you can find how long they were deferred before being executed.
If a search is deferred too long, it will finally be skipped: skipped searches

The root cause are usually caused by :
- too many searches : you have too many searches (or heavy dashboard)
- non optimized scheduled searches taking long to run and overlapping.
- hardware limit : the indexers and search-head have not enough cpu core to handle high search concurrency. (check limits.conf), on 6.0 the formula for historical search concurrency = 6+ 1* (number of cores)

(hint, disable the deployment monitor app if you have it)

Rukmani_Splunk
Path Finder

Thanks a lot

0 Karma

MuS
Legend

more an advice then an answer, check out this answer on search scheduling http://answers.splunk.com/answers/33717/scheduled-searches-for-summary-index-does-not-run-no-skipped...

Get Updates on the Splunk Community!

Now Available: Cisco Talos Threat Intelligence Integrations for Splunk Security Cloud ...

At .conf24, we shared that we were in the process of integrating Cisco Talos threat intelligence into Splunk ...

Preparing your Splunk Environment for OpenSSL3

The Splunk platform will transition to OpenSSL version 3 in a future release. Actions are required to prepare ...

Easily Improve Agent Saturation with the Splunk Add-on for OpenTelemetry Collector

Agent Saturation What and Whys In application performance monitoring, saturation is defined as the total load ...