How recent is recent?

[hulahoop]

On the page 'Manager > Searches and reports,' enabled scheduled searches have a 'View Recent' link. I have 2 scheduled searches running every 5 minutes over the last 5 minutes. Sometimes the 'View Recent' link shows 1 result, sometimes 0 results. I am not seeing anything in splunkd.log to suggest there was a problem executing the search. So I have 2 questions:

• How recent is recent?
• Where can I find the full history of a scheduled search's execution status (in past versions of Splunk, there was once a link to 'View History')?

[Ledion_Bitincka]

In the 4.1 release there is a dashboard that can be used to inspect the state/status of the scheduler and scheduled search history. The view that can be used to check the scheduled search execution can be found in the search app > Status > Scheduler Activity > By Savedsearch

[hulahoop]

Although it doesn't solve the confusion of what happens with the 'View recent' link.

[hulahoop]

Ledion, this is awesome. I found the dashboard. It is exactly what I am looking for. Thank you!

[benstraw · Accepted solution]

The view recent link launches the jobs manager for that search (by adding a savedSearch= parameter. The jobs manager shows all of the searches that are still cached on disk. Jobs are kept on disk by default for 2 periods, a period being the length of time between runs. So for most scheduled searches you will see 2 results in the jobs manager. This setting is dispatch.ttl in savedsearches.conf, but it is not exposed through the ui due to the potential of quickly filling up disk space if this setting is abused.

[hulahoop]

Nope. It is using the default 2p. 😞

[benstraw]

Is your search set to 1p in dispatch.ttl? You can see this setting in savedsearches.conf or by going to the REST endpoint /services/saved/searches. If the results count is 0 there should still be an artifact and it should still show up in the jobs manager. Perhaps there is something with the scheduler that causes an artifact to not be saved if there is no results, I am not sure.

[hulahoop]

Maybe this is because my scheduled search does not actually create any artifacts. The result count is always 0 with no action triggered.

[hulahoop]

Thank you, Ben. So if my scheduled search runs every 5 minutes over the last 5 minutes, does that mean the period is 5 minutes?

If the answer is yes, then this is not what we are seeing. The job in "View recent" disappears in less than 1 period.

[hulahoop]

I will try to answer part 2 of my 2-part question--where to find the full history of a search's execution?

To find the full history, check in the _audit index. Scheduled searches are first granted permission to run, then on completion an audit event is recorded. For each scheduled search executed, these 2 events are written to index=_audit along with a search_id which includes the name of the search.

For example, to get the history of my scheduled search named "Summary - Juniper - Critical NIDS Count" belonging to the App called 'SplunkForJuniperNSM', run the following search:

index="_audit" search_id="scheduler_nobody_SplunkForJuniperNSM_Summary___Juniper___Critical_NIDS_Count*"

This will return 2 events for each execution time:

Audit:[timestamp=02-04-2010 12:50:11.297, user=n/a, action=search, info=completed, search_id="scheduler_nobody_SplunkForJuniperNSMAtUnionBank_Summary___Juniper___Critical_NIDS_Count_at_1265316600_1609417292", total_run_time=0.22 seconds.][n/a]
Audit:[timestamp=02-04-2010 12:50:01.029, user=splunk-system-user, action=search, info=granted , search_id="scheduler_nobody_SplunkForJuniperNSMAtUnionBank_Summary___Juniper___Critical_NIDS_Count_at_1265316600_1609417292", search='search  sourcetype=juniper-nsm-ids Severity=high | sistats count', autojoin=1, buckets=0, ttl=600, max_count=10000, maxtime=0, enable_lookups=1, extra_fields="", apiStartTime="Thu Feb  4 12:45:00 2010", apiEndTime="Thu Feb  4 12:50:00 2010"][n/a]

Maybe there is an easier way to find this info in the Manager, but I haven't uncovered it.