Running Saved Searches with Default Index _internal

[joy76]

HI, I did search query as follows:
index=_internal sourcetype=scheduler

And I get to see things in the resulting log that certain fields appears such as "run_time", "dispatch_time", and
"scheduled_time."

I want to know every nitty-gritty details in how those certain fields working inside and outside.
You could pinpoint where I can find those info or you could explain in your own words fully here.
Especially, I would like to understand completely in inner mechanism of "dispatch_time."

Thanks.

[kristian_kolb · Accepted solution]

Here goes;

scheduled_time = the point in time when a scheduled search was destined to run. Expressed in epoch (number of seconds since midnight (00:00:00 AM) Jan 1 1970). This will be a new value for each scheduled run of a particular search. If you have a search that's scheduled to run every hour, you will see that the difference in the scheduled_time in the scheduler.log will be 3600 for consecutive occurrences of that search.

dispatch_time = the actual time the search was started (also in epoch). Set in relation to the scheduled_time it will indicate if the searches start at the time they are supposed to.

run_time = how long time it took to execute the search (in seconds)

A large difference (more than a few seconds) between scheduled_time and dispatch_time could suggest that you have too many searches scheduled to start at the same time (so some of them are queued up). Or that the system is overloaded with manual searches (or is generally overloaded).

---

UPDATE:

Note that Splunk may decide to skip a search before the next scheduled run - perhaps if it has not been able to launch the search in 5/15/30 minutes - I just don't know the default timeout settings. However, I'm not sure when Splunk decides to skip a search and move on. Somebody more knowledgeable will have to answer that.

EDIT: removed some faulty assumptions.

/K

[kristian_kolb]

Hi, joy76, I seem to have made some assumptions regarding this, that did not turn out right. It seems like searches are not skipped when the next scheduled run is due. Sorry.

[joy76]

Kristian, regarding setting "dispatch_time" in relation with "scheduled_time".., when scheduled searches (e.g. S0~S24) are scheduled to be run at the same time, some of them are queued up.
So, some scheduled searches(e.g. S20~S24) which have not been dispatched on their scheduled times will go into QUEUE list, waiting. Then, how will I check to see if how many number of searches are currently being queued up in the list and in what order are they queued up in the list OR which saved search(es) are they in the list ??

[kristian_kolb]

update above, in the original answer.

[kristian_kolb]

I mean that I updated my original answer, just scroll up this page a little bit 🙂

[kristian_kolb]

Some more info in the answer. /K

[piebob]

Joy, if this Answer solved your problem, please accept Kristian's Answer by clicking the checkbox. thanks!