Splunk Search

When does a job get skipped and does it ever re-run?

a212830
Champion

Hi,

I've had some complaints lately about jobs not running. A couple of questions...

1) How can I validate if a specific job was skipped?
2) Do skipped jobs ever re-run?
3) How can I monitor all my skipped jobs?
4) For summary indexing, how do I recover that data?

Thanks a bunch! I will be looking at the doc for these questions, but, as you know, Splunk almost has too much doc!

1 Solution

somesoni2
Revered Legend

Here are the answers

1) You can check the scheduler log for a search to check if it was skipped.

index=_internal sourcetype=scheduler status=skipped savedsearch_name="YourSavedSearch"

2) By default the skipped saved search will not run. This is due to by default the scheduler computes the next execution time based on the current time. So what's skipped before now is skipped. But there is an option in savedsearches.conf (not available from Splunk Web, need to edit savedsearches.conf for each search/or set is globally), called realtime_schedule. By default it's 1 (means next schedule is based on current time). Setting it to 0 will force scheduler to compute the next execution based on last search execution time (also called as continuous scheduling). So, any skipped searches will be re-run till it catches on. See more details here. http://docs.splunk.com/Documentation/Splunk/6.2.6/Admin/Savedsearchesconf

3) You can use the same search, provided in answer 1 and do you monitoring around that. May be something like this

index=_internal sourcetypye=scheduler status=skipped | stats count by savedsearch_name

4) If the realtime_schedule=0, the summary indexing will backfill itself for gaps. If that was not used, you need to manually backfill the summary index search. See here http://docs.splunk.com/Documentation/Splunk/6.0.2/Knowledge/Managesummaryindexgapsandoverlaps

Hope this helps.

View solution in original post

somesoni2
Revered Legend

Here are the answers

1) You can check the scheduler log for a search to check if it was skipped.

index=_internal sourcetype=scheduler status=skipped savedsearch_name="YourSavedSearch"

2) By default the skipped saved search will not run. This is due to by default the scheduler computes the next execution time based on the current time. So what's skipped before now is skipped. But there is an option in savedsearches.conf (not available from Splunk Web, need to edit savedsearches.conf for each search/or set is globally), called realtime_schedule. By default it's 1 (means next schedule is based on current time). Setting it to 0 will force scheduler to compute the next execution based on last search execution time (also called as continuous scheduling). So, any skipped searches will be re-run till it catches on. See more details here. http://docs.splunk.com/Documentation/Splunk/6.2.6/Admin/Savedsearchesconf

3) You can use the same search, provided in answer 1 and do you monitoring around that. May be something like this

index=_internal sourcetypye=scheduler status=skipped | stats count by savedsearch_name

4) If the realtime_schedule=0, the summary indexing will backfill itself for gaps. If that was not used, you need to manually backfill the summary index search. See here http://docs.splunk.com/Documentation/Splunk/6.0.2/Knowledge/Managesummaryindexgapsandoverlaps

Hope this helps.

a212830
Champion

Awesome. Thanks!

0 Karma

sloshburch
Splunk Employee
Splunk Employee

Don't forget about the DMC (or SoS) which should show it as well. Since I have a SHC set up, it shows in the shc_scheduler_delegation_statistics dashboard. If you don't have SHC it may be hidden in some type of Scheduler Activity panel on another dashboard.

0 Karma
Get Updates on the Splunk Community!

The Splunk Success Framework: Your Guide to Successful Splunk Implementations

Splunk Lantern is a customer success center that provides advice from Splunk experts on valuable data ...

Splunk Training for All: Meet Aspiring Cybersecurity Analyst, Marc Alicea

Splunk Education believes in the value of training and certification in today’s rapidly-changing data-driven ...

Investigate Security and Threat Detection with VirusTotal and Splunk Integration

As security threats and their complexities surge, security analysts deal with increased challenges and ...