Alerting

Is it possible to purposely cause a scheduled search to be skipped?

ben_leung
Builder

This is for testing an alert to see when scheduled searches are skipped, causing the logs to write status=skipped instead of status=success OR status=continued.

Tags (2)
0 Karma
1 Solution

jrodman
Splunk Employee
Splunk Employee

I don't believe we have this functionality (force skip for testing).
Scheduled searches are skipped when they are still running at the end of the interval, so if you schedule a search to run every 5 minutes, for example, and arrange to have the search take longer than that, it will be skipped.

If you have a large index, it isn't too difficult to write a very slow search, doing something like retreiving a very large number of events, adding a |noop local then sorting the results repeatedly on different fields.

Another option would be to add a custom python search command that you can make arbitrarily slow.

View solution in original post

woodcock
Esteemed Legend

You can conditionally short-circuit any scheduled search job by using addinfo and map for the blackout logic that it is not supposed to run like this (extra steps for clarity):

... | dedup status| status="skipped" | stats count AS blackout | addinfo | eval blackout= if(blackout>0,"YES","NO") | eval earliestMaybe=if((blackout=="NO"), info_min_time, now()) | map search="search earliest=$earliestMaybe$ latest=$info_max_time$ YOUR SEARCH HERE"

For the condition that it is not supposed to run (if the last event has status = "skipped"), the search will generate an error.

Here are other similar questions with answers that should work for you (this answer is derived from the first one):

http://answers.splunk.com/answering/260370/view.html

http://answers.splunk.com/answers/24824/can-i-set-a-blackout-period-for-a-scheduled-search-during-wh...

jrodman
Splunk Employee
Splunk Employee

I don't believe we have this functionality (force skip for testing).
Scheduled searches are skipped when they are still running at the end of the interval, so if you schedule a search to run every 5 minutes, for example, and arrange to have the search take longer than that, it will be skipped.

If you have a large index, it isn't too difficult to write a very slow search, doing something like retreiving a very large number of events, adding a |noop local then sorting the results repeatedly on different fields.

Another option would be to add a custom python search command that you can make arbitrarily slow.

ben_leung
Builder

Makes sense to make the search pass the internal time range. Testing this would be easy, but the actual query that I wanted to test runs every 15 minutes that completes relatively quick, within a minute. Thanks, I will complete my testing with this information provided.

0 Karma
Get Updates on the Splunk Community!

Announcing Scheduled Export GA for Dashboard Studio

We're excited to announce the general availability of Scheduled Export for Dashboard Studio. Starting in ...

Extending Observability Content to Splunk Cloud

Watch Now!   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to leverage ...

More Control Over Your Monitoring Costs with Archived Metrics GA in US-AWS!

What if there was a way you could keep all the metrics data you need while saving on storage costs?This is now ...