Alerting

Is it possible to purposely cause a scheduled search to be skipped?

Builder

This is for testing an alert to see when scheduled searches are skipped, causing the logs to write status=skipped instead of status=success OR status=continued.

Tags (2)
0 Karma
1 Solution

Splunk Employee
Splunk Employee

I don't believe we have this functionality (force skip for testing).
Scheduled searches are skipped when they are still running at the end of the interval, so if you schedule a search to run every 5 minutes, for example, and arrange to have the search take longer than that, it will be skipped.

If you have a large index, it isn't too difficult to write a very slow search, doing something like retreiving a very large number of events, adding a |noop local then sorting the results repeatedly on different fields.

Another option would be to add a custom python search command that you can make arbitrarily slow.

View solution in original post

Esteemed Legend

You can conditionally short-circuit any scheduled search job by using addinfo and map for the blackout logic that it is not supposed to run like this (extra steps for clarity):

... | dedup status| status="skipped" | stats count AS blackout | addinfo | eval blackout= if(blackout>0,"YES","NO") | eval earliestMaybe=if((blackout=="NO"), info_min_time, now()) | map search="search earliest=$earliestMaybe$ latest=$info_max_time$ YOUR SEARCH HERE"

For the condition that it is not supposed to run (if the last event has status = "skipped"), the search will generate an error.

Here are other similar questions with answers that should work for you (this answer is derived from the first one):

http://answers.splunk.com/answering/260370/view.html

http://answers.splunk.com/answers/24824/can-i-set-a-blackout-period-for-a-scheduled-search-during-wh...

Splunk Employee
Splunk Employee

I don't believe we have this functionality (force skip for testing).
Scheduled searches are skipped when they are still running at the end of the interval, so if you schedule a search to run every 5 minutes, for example, and arrange to have the search take longer than that, it will be skipped.

If you have a large index, it isn't too difficult to write a very slow search, doing something like retreiving a very large number of events, adding a |noop local then sorting the results repeatedly on different fields.

Another option would be to add a custom python search command that you can make arbitrarily slow.

View solution in original post

Builder

Makes sense to make the search pass the internal time range. Testing this would be easy, but the actual query that I wanted to test runs every 15 minutes that completes relatively quick, within a minute. Thanks, I will complete my testing with this information provided.

0 Karma
State of Splunk Careers

Access the Splunk Careers Report to see real data that shows how Splunk mastery increases your value and job satisfaction.

Find out what your skills are worth!