When does a job get skipped and does it ever re-run?

[a212830]

Hi,

I've had some complaints lately about jobs not running. A couple of questions...

1) How can I validate if a specific job was skipped?
2) Do skipped jobs ever re-run?
3) How can I monitor all my skipped jobs?
4) For summary indexing, how do I recover that data?

Thanks a bunch! I will be looking at the doc for these questions, but, as you know, Splunk almost has too much doc!

[somesoni2 · Accepted solution]

Here are the answers

1) You can check the scheduler log for a search to check if it was skipped.

index=_internal sourcetype=scheduler status=skipped savedsearch_name="YourSavedSearch"

2) By default the skipped saved search will not run. This is due to by default the scheduler computes the next execution time based on the current time. So what's skipped before now is skipped. But there is an option in savedsearches.conf (not available from Splunk Web, need to edit savedsearches.conf for each search/or set is globally), called realtime_schedule. By default it's 1 (means next schedule is based on current time). Setting it to 0 will force scheduler to compute the next execution based on last search execution time (also called as continuous scheduling). So, any skipped searches will be re-run till it catches on. See more details here. http://docs.splunk.com/Documentation/Splunk/6.2.6/Admin/Savedsearchesconf

3) You can use the same search, provided in answer 1 and do you monitoring around that. May be something like this

index=_internal sourcetypye=scheduler status=skipped | stats count by savedsearch_name

4) If the realtime_schedule=0, the summary indexing will backfill itself for gaps. If that was not used, you need to manually backfill the summary index search. See here http://docs.splunk.com/Documentation/Splunk/6.0.2/Knowledge/Managesummaryindexgapsandoverlaps

Hope this helps.

[a212830]

Awesome. Thanks!

[sloshburch]

Don't forget about the DMC (or SoS) which should show it as well. Since I have a SHC set up, it shows in the shc_scheduler_delegation_statistics dashboard. If you don't have SHC it may be hidden in some type of Scheduler Activity panel on another dashboard.