16-59/10 5-6 * * * cron was setup for more than 0 events.
We had an event at 5:15 Am. Any idea why the alert did not trigger?
The query used is for -5m@m
With that cron schedule, I guess the search ran first time at 5:20 AM? Did you confirm the search actually ran, and indeed ran at that time?
According to me,cron expression = 16-59/10 5-6 * * * means the search query will trigger at 5 hours and between 16 to 59 minutes in a span of 10 minutes, same for the hour 6.
So it will run,
5:16, 5:26, 5:36, 5:46, 5:56 and same for 6th hour
Hmm, I might be wrong about that then. I also checked with crontab guru and that agrees with you that it would run at 16,26,36,46,56 : https://crontab.guru/#16-59/10_5-6_*_*_*
Note: I added 2 stars at the end to make it a proper complete cron schedule.
From the settings page for saved searches, you should see a "View Recent" link in the actions column. Which allows you to inspect recent search executions. Also saved search executions are logged in index=_audit.
Just because your event happened at that time does not mean that it was indexed and searchable at the time the search ran. A window so short as "within the last 5 minutes" leaves very little time for pipeline latencies which are common forwarding events into Splunk. If you compare the value of
_indextime for that event and they are more than 5-minutes apart (300 seconds), then the latency indicates that the event was not searchable in Splunk when the search looking for it ran.