On the page 'Manager > Searches and reports,' enabled scheduled searches have a 'View Recent' link. I have 2 scheduled searches running every 5 minutes over the last 5 minutes. Sometimes the 'View Recent' link shows 1 result, sometimes 0 results. I am not seeing anything in splunkd.log to suggest there was a problem executing the search. So I have 2 questions:
I will try to answer part 2 of my 2-part question--where to find the full history of a search's execution?
To find the full history, check in the
_audit index. Scheduled searches are first granted permission to run, then on completion an audit event is recorded. For each scheduled search executed, these 2 events are written to
index=_audit along with a
search_id which includes the name of the search.
For example, to get the history of my scheduled search named "Summary - Juniper - Critical NIDS Count" belonging to the App called 'SplunkForJuniperNSM', run the following search:
This will return 2 events for each execution time:
Audit:[timestamp=02-04-2010 12:50:11.297, user=n/a, action=search, info=completed, search_id="scheduler_nobody_SplunkForJuniperNSMAtUnionBank_Summary___Juniper___Critical_NIDS_Count_at_1265316600_1609417292", total_run_time=0.22 seconds.][n/a] Audit:[timestamp=02-04-2010 12:50:01.029, user=splunk-system-user, action=search, info=granted , search_id="scheduler_nobody_SplunkForJuniperNSMAtUnionBank_Summary___Juniper___Critical_NIDS_Count_at_1265316600_1609417292", search='search sourcetype=juniper-nsm-ids Severity=high | sistats count', autojoin=1, buckets=0, ttl=600, max_count=10000, maxtime=0, enable_lookups=1, extra_fields="", apiStartTime="Thu Feb 4 12:45:00 2010", apiEndTime="Thu Feb 4 12:50:00 2010"][n/a]
Maybe there is an easier way to find this info in the Manager, but I haven't uncovered it.
The view recent link launches the jobs manager for that search (by adding a savedSearch= parameter. The jobs manager shows all of the searches that are still cached on disk. Jobs are kept on disk by default for 2 periods, a period being the length of time between runs. So for most scheduled searches you will see 2 results in the jobs manager. This setting is dispatch.ttl in savedsearches.conf, but it is not exposed through the ui due to the potential of quickly filling up disk space if this setting is abused.
Thank you, Ben. So if my scheduled search runs every 5 minutes over the last 5 minutes, does that mean the period is 5 minutes?
If the answer is yes, then this is not what we are seeing. The job in "View recent" disappears in less than 1 period.
Maybe this is because my scheduled search does not actually create any artifacts. The result count is always 0 with no action triggered.
Is your search set to 1p in dispatch.ttl? You can see this setting in savedsearches.conf or by going to the REST endpoint
In the 4.1 release there is a dashboard that can be used to inspect the state/status of the scheduler and scheduled search history. The view that can be used to check the scheduled search execution can be found in the search app > Status > Scheduler Activity > By Savedsearch