topic Re: Why the alert did not trigger for below cron expression? in Alerting

Why the alert did not trigger for below cron expression?

abhi04 — Fri, 29 Jun 2018 10:02:26 GMT

16-59/10 5-6 * * * cron was setup for more than 0 events.

We had an event at 5:15 Am. Any idea why the alert did not trigger?

The query used is for -5m@m

Re: Why the alert did not trigger for below cron expression?

FrankVl — Fri, 29 Jun 2018 10:18:56 GMT

With that cron schedule, I guess the search ran first time at 5:20 AM? Did you confirm the search actually ran, and indeed ran at that time?

Re: Why the alert did not trigger for below cron expression?

abhi04 — Fri, 29 Jun 2018 10:52:07 GMT

@FrankVI
Should not the search run at 5:16 and check for last 5 minutes? Also, how to check when the search ran at that time?

Re: Why the alert did not trigger for below cron expression?

FrankVl — Fri, 29 Jun 2018 10:58:45 GMT

No, you set it to /10, so it runs at 0,10,20,30,40,50 (where 0 and 10 are skipped because of your 16-59 time window).

Re: Why the alert did not trigger for below cron expression?

abhi04 — Fri, 29 Jun 2018 11:01:30 GMT

I just checked and confirmed that the it is scheduled 05:16:00

Re: Why the alert did not trigger for below cron expression?

abhi04 — Fri, 29 Jun 2018 11:05:33 GMT

According to me,cron expression = 16-59/10 5-6 * * * means the search query will trigger at 5 hours and between 16 to 59 minutes in a span of 10 minutes, same for the hour 6.

So it will run,

5:16, 5:26, 5:36, 5:46, 5:56 and same for 6th hour

Re: Why the alert did not trigger for below cron expression?

FrankVl — Fri, 29 Jun 2018 11:48:17 GMT

Hmm, I might be wrong about that then. I also checked with crontab guru and that agrees with you that it would run at 16,26,36,46,56 : https://crontab.guru/#16-59/10_5-6_*_*_*

Note: I added 2 stars at the end to make it a proper complete cron schedule.

From the settings page for saved searches, you should see a "View Recent" link in the actions column. Which allows you to inspect recent search executions. Also saved search executions are logged in index=_audit.

Re: Why the alert did not trigger for below cron expression?

woodcock — Sat, 30 Jun 2018 03:29:53 GMT

Just because your event happened at that time does not mean that it was indexed and searchable at the time the search ran. A window so short as "within the last 5 minutes" leaves very little time for pipeline latencies which are common forwarding events into Splunk. If you compare the value of _time with _indextime for that event and they are more than 5-minutes apart (300 seconds), then the latency indicates that the event was not searchable in Splunk when the search looking for it ran.

Re: Why the alert did not trigger for below cron expression?

woodcock — Sat, 30 Jun 2018 03:30:20 GMT

And before @mattymo says, it: Meta W00t!