Tere is an app wich can help you manage your alerts as you need: http://challengepost.com/software/alert-manager
Concerning my own researches here is what i was able to do.
First, notice that the _internal index let you manage your alerts. For example, you can get your alert thread_id which can be AlertNotifierWorker-0 AlertNotifierWorker-1,.......................
Each alert has a unique AlertNotifierWorker, which is increment each time the alert is triggered.
Now let suppose that you have two alerts.
Alert1 with thread_id=AlertNotifierWorker-0 and currentcount0 as the current count of alerts triggered,
Alert2 with thread_id=AlertNotifierWorker-2 and currentcount2 as the current count of alerts triggered.
suppositions:
1. We have schedulded Alert1 to triggered at 5 min window
2. We have scheduled Alert2 to triggered at 1 min window, only if Alert1 is triggered.
3. the started time is 6 am.
4. Alert1 is always triggered
Conditions:
Alert1 is triggered when the word "error" is seen for the last 5min in your events.
Alert2 if Alert1 is triggered, means, currentcount0>currentcount2.
initialisations:
if you
Algorithm
At 6H:5min am, Alert1 is triggered, AlertNotifierWorker-0 is increment, means AlertNotifierWorker-0==1
At 6H:6min am, Alert2 is triggered, because AlertNotifierWorker-0>AlertNotifierWorker-2. AlertNotifierWorker-2 is then increment, means AlertNotifierWorker-2==1.
From 6H:6min To 6H: 9min, Since AlertNotifierWorker-0==AlertNotifierWorker-2, no alert is triggered.
At 6H:10min, Alert1 is triggered, means AlertNotifierWorker-0==2
At 6H:11min am, Alert2 is triggered, because AlertNotifierWorker-0>AlertNotifierWorker-2. AlertNotifierWorker-2 is then increment, means AlertNotifierWorker-2==2.
From 6H:6min To 6H: 9min, Since AlertNotifierWorker-0==AlertNotifierWorker-2, no alert is triggered.
.........................
...............................and so on
implementation
Alert1:
search query: index=* OR index=_* "error"
Launch the search and save it as an alert, and set it as described above
Alert2:
Search qury:
index=_internal sourcetype=scheduler thread_id=AlertNotifier* NOT (alert_actions="summary_index" OR alert_actions="") thread_id="AlertNotifierWorker-0"|stats count(thread_id) as currentval|join [search index=_internal sourcetype=scheduler thread_id=AlertNotifier* NOT (alert_actions="summary_index" OR alert_actions="") thread_id="AlertNotifierWorker-2"|stats count(thread_id) as currentval2 ] |where currentval>currentval2 |stats values(currentval) values(currentval2)
Set it to triggered as said above, and let me know if any questions.
... View more