I have a strange situation with the delays in both scheduling and dispatching of my alerts.
They should run each minute, as per cron schedule:
*/1 * * * *
but, when I am checking the schedule and dispatch times I can see that:
1/ The alerts get scheduled each second minute only
2/ There is always a delay between the schedule and dispatch, more less always 2 minutes as well, please see the attached image.
Could you please advise what's going wrong here?
How would I get my alerts executed each minute and get rid of the additional delay between schedule and dispatch?
I thought that the schedule to dispatch delay could come from the resource bottleneck, but there is none.
Also, the fact that it is always 2 minutes would not fit in the resource bottleneck theory.
Are there any parameters that could cause the above behavior?
since this sounds like some config is actually telling splunk to wait that 2 minutes you were talking about, I suggest you may
This user is providing knowledge about the schedule_window field for sheduled searches. Might be something you want to check.
Unfortunately it did not help. The action I took as per the description in link was to grant explicitly the editsearchschedulewindow role to my user in order to get the schedulewindow = 0 and not default.
It did not help. I can see all of my and not only my alerts to have a lag of precisely 2 minutes. This is strange, because there are still some other alerts in the system that get dispatched immediately. When I compare the parameters of the both in the system, they seem the same.
1/ my alert with the 2 min lag:
01-23-2019 13:29:15.239 +0100 INFO SavedSplunker - savedsearch_id="nobody;mlbso;Anomaly Detection", search_type="scheduled", user="CDE", app="mlbso", savedsearch_name="Anomaly Detection", priority=default, status=success, digest_mode=1, scheduled_time=1548246420, window_time=0, dispatch_time=1548246548, run_time=5.707, result_count=0, alert_actions="", sid="scheduler__CDE__mlbso__RMD54eeec7fba2d5a846_at_1548246420_4375", suppressed=0, thread_id="AlertNotifierWorker-0"
other alert, dispatched immediately (without lag):
01-23-2019 12:35:01.097 +0000 INFO SavedSplunker - savedsearch_id="nobody;ids;sci_prod_us_east http 5xx", search_type="scheduled", user="ABC", app="ids", savedsearch_name="sci_prod_us_east http 5xx", priority=default, status=success, digest_mode=1, scheduled_time=1548246900, window_time=0, dispatch_time=1548246900, run_time=0.235, result_count=0, alert_actions="", sid="scheduler__ABC__ids__RMD5494dd652a11e08f4_at_1548246900_25299", suppressed=0, thread_id="AlertNotifierWorker-0"
Could you advise?
Is there any way to see the detailed scheduler log for this issueing a search in Splunk?
Waht would be the reason to have this kind of lag?
In the scheduler log there is nothing more than the entries as above 1/ and 2/. I am not able to figure out where the lag comes from based on it.
Any further ideas?