Splunk Search

Skipped Searches

g_paternicola
Path Finder

Hi everyone, I have some questions about skipped searches. With the following search, I have found, that on my SH I have a few (2800 last 7 days) skipped searches. 

 

index = _internal skipped sourcetype=scheduler status=skipped
| stats count by app search_type reason savedsearch_name 
| sort -count

 


I have made other searches with show me all saved searches and their scheduled cronjob. I have found, that I have more than 70 searches that are running every 5 minutes and a few are running every minute. 

Would that be my issue with the skipped searches, even they are running for just a few seconds (max 5 seconds). On all 70 scheduled searches is the parameter schedule_window set to 0.

Labels (2)
0 Karma
1 Solution

richgalloway
SplunkTrust
SplunkTrust

The 'auto' calculation accounts for the search interval and will not make a 5-minute search wait 15 minutes.  If you're not comfortable with 'auto' then use a specific number (other than zero).  

---
If this reply helps you, Karma would be appreciated.

View solution in original post

richgalloway
SplunkTrust
SplunkTrust

If there are 70 searches scheduled to run at the same time, fewer than 70 available search "slots", and schedule_window=0 then some of the 70 will be skipped.  It doesn't matter how quickly the searches run because the decision to skip has already been made.

The easiest (and best, IMO) way to avoid skipped searches is to set schedule_window=auto.  This allows the scheduler to wait briefly to see if a CPU becomes available before deciding to skip a search.  

Another good way to avoid skipped searches is to distribute search times evenly around the hour.  Hourly searches don't have to run at minute zero, for example.  Daily searches don't have to run exactly at midnight.  Few searches need to run every minute.

---
If this reply helps you, Karma would be appreciated.
0 Karma

g_paternicola
Path Finder

Yeah, this makes absolute sense to me, but It also depends on how the "auto" calculation works. if that could generate a delay of 15 mins then it makes no sense for the */5 and */1 searches. 

Do you know how the calculation works?

Tags (2)
0 Karma

richgalloway
SplunkTrust
SplunkTrust

The 'auto' calculation accounts for the search interval and will not make a 5-minute search wait 15 minutes.  If you're not comfortable with 'auto' then use a specific number (other than zero).  

---
If this reply helps you, Karma would be appreciated.
Get Updates on the Splunk Community!

Introduction to Splunk Observability Cloud - Building a Resilient Hybrid Cloud

Introduction to Splunk Observability Cloud - Building a Resilient Hybrid Cloud  In today’s fast-paced digital ...

Observability protocols to know about

Observability protocols define the specifications or formats for collecting, encoding, transporting, and ...

Take Your Breath Away with Splunk Risk-Based Alerting (RBA)

WATCH NOW!The Splunk Guide to Risk-Based Alerting is here to empower your SOC like never before. Join Haylee ...