We have a rare query from a team and situation is
- The team needs to immediately get an alert (within 5 minutes)
- The team don't want to miss an alert (even if there is a network delay or indexing delay or some issues)
- Also the alert cannot be duplicated if it is already alerted
- Alerting is done on the event-time (and not indextime)
So an event comes in real-time, then there is no problem.
But if there are network issues, and assume the event got delayed by 6 minutes they need a sweep up of such delayed alert, so it will be still alerted, but NOT shown duplicate
I thought of an option to
- have two searches (SavedSearch1 => one which runs every 5mins, searching for previous 5 mins) & (SavedSearch2=> which runs every 5 mins but sweeps events in last 60mins).
- Summary index the SavedSearch1 and SavedSearch2 should compare if it is already there in the summary index using same time
So my query is
Have you guys done this similar situation? Is there any other better option?
and why the following wont work?
_index_earliest=-4m@m _index_latest=-1m@m .... all your query ... | stats <whatever> by _time _indextime ... | where or eval if needed ...
then run the alert every 3 minutes and add your relevant condition in the alert dialog boxes.
you are not supposed to miss a thing
oh, dont forget to mark the search / alert as highest priority
hope it helps