topic Re: Skipped Searches in Splunk Search

Skipped Searches

g_paternicola — Fri, 13 Aug 2021 13:02:22 GMT

Hi everyone, I have some questions about skipped searches. With the following search, I have found, that on my SH I have a few (2800 last 7 days) skipped searches.

index = _internal skipped sourcetype=scheduler status=skipped | stats count by app search_type reason savedsearch_name | sort -count

I have made other searches with show me all saved searches and their scheduled cronjob. I have found, that I have more than 70 searches that are running every 5 minutes and a few are running every minute.

Would that be my issue with the skipped searches, even they are running for just a few seconds (max 5 seconds). On all 70 scheduled searches is the parameter schedule_window set to 0.

Re: Skipped Searches

richgalloway — Fri, 13 Aug 2021 13:29:52 GMT

If there are 70 searches scheduled to run at the same time, fewer than 70 available search "slots", and schedule_window=0 then some of the 70 will be skipped. It doesn't matter how quickly the searches run because the decision to skip has already been made.

The easiest (and best, IMO) way to avoid skipped searches is to set schedule_window=auto. This allows the scheduler to wait briefly to see if a CPU becomes available before deciding to skip a search.

Another good way to avoid skipped searches is to distribute search times evenly around the hour. Hourly searches don't have to run at minute zero, for example. Daily searches don't have to run exactly at midnight. Few searches need to run every minute.

Re: Skipped Searches

g_paternicola — Fri, 13 Aug 2021 13:49:22 GMT

Yeah, this makes absolute sense to me, but It also depends on how the "auto" calculation works. if that could generate a delay of 15 mins then it makes no sense for the */5 and */1 searches.

Do you know how the calculation works?

Re: Skipped Searches

richgalloway — Fri, 13 Aug 2021 16:29:32 GMT

The 'auto' calculation accounts for the search interval and will not make a 5-minute search wait 15 minutes. If you're not comfortable with 'auto' then use a specific number (other than zero).