Alerting

Why the alert did not trigger for below cron expression?

abhi04
Communicator

16-59/10 5-6 * * * cron was setup for more than 0 events.

We had an event at 5:15 Am. Any idea why the alert did not trigger?

The query used is for -5m@m

Tags (1)
0 Karma

woodcock
Esteemed Legend

Just because your event happened at that time does not mean that it was indexed and searchable at the time the search ran. A window so short as "within the last 5 minutes" leaves very little time for pipeline latencies which are common forwarding events into Splunk. If you compare the value of _time with _indextime for that event and they are more than 5-minutes apart (300 seconds), then the latency indicates that the event was not searchable in Splunk when the search looking for it ran.

0 Karma

woodcock
Esteemed Legend

And before @mattymo says, it: Meta W00t!

0 Karma

FrankVl
Ultra Champion

With that cron schedule, I guess the search ran first time at 5:20 AM? Did you confirm the search actually ran, and indeed ran at that time?

0 Karma

abhi04
Communicator

@FrankVI
Should not the search run at 5:16 and check for last 5 minutes? Also, how to check when the search ran at that time?

0 Karma

abhi04
Communicator

I just checked and confirmed that the it is scheduled 05:16:00

0 Karma

FrankVl
Ultra Champion

Hmm, I might be wrong about that then. I also checked with crontab guru and that agrees with you that it would run at 16,26,36,46,56 : https://crontab.guru/#16-59/10_5-6_*_*_*

Note: I added 2 stars at the end to make it a proper complete cron schedule.

From the settings page for saved searches, you should see a "View Recent" link in the actions column. Which allows you to inspect recent search executions. Also saved search executions are logged in index=_audit.

0 Karma

FrankVl
Ultra Champion

No, you set it to /10, so it runs at 0,10,20,30,40,50 (where 0 and 10 are skipped because of your 16-59 time window).

0 Karma

abhi04
Communicator

According to me,cron expression = 16-59/10 5-6 * * * means the search query will trigger at 5 hours and between 16 to 59 minutes in a span of 10 minutes, same for the hour 6.

So it will run,

5:16, 5:26, 5:36, 5:46, 5:56 and same for 6th hour

0 Karma
Get Updates on the Splunk Community!

Splunk Observability as Code: From Zero to Dashboard

For the details on what Self-Service Observability and Observability as Code is, we have some awesome content ...

[Puzzles] Solve, Learn, Repeat: Character substitutions with Regular Expressions

This challenge was first posted on Slack #puzzles channelFor BORE at .conf23, we had a puzzle question which ...

Shape the Future of Splunk: Join the Product Research Lab!

Join the Splunk Product Research Lab and connect with us in the Slack channel #product-research-lab to get ...