Splunk Search

Skipped Searches

g_paternicola
Path Finder

Hi everyone, I have some questions about skipped searches. With the following search, I have found, that on my SH I have a few (2800 last 7 days) skipped searches. 

 

index = _internal skipped sourcetype=scheduler status=skipped
| stats count by app search_type reason savedsearch_name 
| sort -count

 


I have made other searches with show me all saved searches and their scheduled cronjob. I have found, that I have more than 70 searches that are running every 5 minutes and a few are running every minute. 

Would that be my issue with the skipped searches, even they are running for just a few seconds (max 5 seconds). On all 70 scheduled searches is the parameter schedule_window set to 0.

Labels (2)
0 Karma
1 Solution

richgalloway
SplunkTrust
SplunkTrust

The 'auto' calculation accounts for the search interval and will not make a 5-minute search wait 15 minutes.  If you're not comfortable with 'auto' then use a specific number (other than zero).  

---
If this reply helps you, Karma would be appreciated.

View solution in original post

richgalloway
SplunkTrust
SplunkTrust

If there are 70 searches scheduled to run at the same time, fewer than 70 available search "slots", and schedule_window=0 then some of the 70 will be skipped.  It doesn't matter how quickly the searches run because the decision to skip has already been made.

The easiest (and best, IMO) way to avoid skipped searches is to set schedule_window=auto.  This allows the scheduler to wait briefly to see if a CPU becomes available before deciding to skip a search.  

Another good way to avoid skipped searches is to distribute search times evenly around the hour.  Hourly searches don't have to run at minute zero, for example.  Daily searches don't have to run exactly at midnight.  Few searches need to run every minute.

---
If this reply helps you, Karma would be appreciated.
0 Karma

g_paternicola
Path Finder

Yeah, this makes absolute sense to me, but It also depends on how the "auto" calculation works. if that could generate a delay of 15 mins then it makes no sense for the */5 and */1 searches. 

Do you know how the calculation works?

Tags (2)
0 Karma

richgalloway
SplunkTrust
SplunkTrust

The 'auto' calculation accounts for the search interval and will not make a 5-minute search wait 15 minutes.  If you're not comfortable with 'auto' then use a specific number (other than zero).  

---
If this reply helps you, Karma would be appreciated.
Get Updates on the Splunk Community!

Enterprise Security Content Update (ESCU) | New Releases

In November, the Splunk Threat Research Team had one release of new security content via the Enterprise ...

Index This | Divide 100 by half. What do you get?

November 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with this ...

Stay Connected: Your Guide to December Tech Talks, Office Hours, and Webinars!

❄️ Celebrate the season with our December lineup of Community Office Hours, Tech Talks, and Webinars! ...