Is it possible to create an alert that depends on another alert to be triggered?

[vinodmadaan]

Hi Guys,

I am asking this question out of curiosity (don't even know if this is possible!).

The question is: Is it possible to create an alert that depends on another alert? For example, if Alert-1 is triggered then only Alert-2 will be triggered (point to note is that : both use different searches, so the option that both can have same search is out)?

Thanks in advance,
Vinod.

[stephanefotso · Accepted solution]

Tere is an app wich can help you manage your alerts as you need: http://challengepost.com/software/alert-manager
Concerning my own researches here is what i was able to do.

First, notice that the _internal index let you manage your alerts. For example, you can get your alert thread_id which can be AlertNotifierWorker-0 AlertNotifierWorker-1,.......................
Each alert has a unique AlertNotifierWorker, which is increment each time the alert is triggered.
Now let suppose that you have two alerts.
Alert1 with thread_id=AlertNotifierWorker-0 and currentcount0 as the current count of alerts triggered,
Alert2 with thread_id=AlertNotifierWorker-2 and currentcount2 as the current count of alerts triggered.
suppositions:
1. We have schedulded Alert1 to triggered at 5 min window
2. We have scheduled Alert2 to triggered at 1 min window, only if Alert1 is triggered.
3. the started time is 6 am.
4. Alert1 is always triggered

Conditions:
Alert1 is triggered when the word "error" is seen for the last 5min in your events.
Alert2 if Alert1 is triggered, means, currentcount0>currentcount2.

initialisations:
if you

Algorithm

1. At 6H:5min am, Alert1 is triggered, AlertNotifierWorker-0 is increment, means AlertNotifierWorker-0==1
2. At 6H:6min am, Alert2 is triggered, because AlertNotifierWorker-0>AlertNotifierWorker-2. AlertNotifierWorker-2 is then increment, means AlertNotifierWorker-2==1.
3. From 6H:6min To 6H: 9min, Since AlertNotifierWorker-0==AlertNotifierWorker-2, no alert is triggered.
4. At 6H:10min, Alert1 is triggered, means AlertNotifierWorker-0==2
5. At 6H:11min am, Alert2 is triggered, because AlertNotifierWorker-0>AlertNotifierWorker-2. AlertNotifierWorker-2 is then increment, means AlertNotifierWorker-2==2.
6. From 6H:6min To 6H: 9min, Since AlertNotifierWorker-0==AlertNotifierWorker-2, no alert is triggered.
7. .........................
8. ...............................and so on

implementation

Alert1:
search query: index=* OR index=_* "error"
Launch the search and save it as an alert, and set it as described above

Alert2:
Search qury:

index=_internal sourcetype=scheduler thread_id=AlertNotifier* NOT (alert_actions="summary_index" OR alert_actions="") thread_id="AlertNotifierWorker-0"|stats count(thread_id) as currentval|join [search index=_internal sourcetype=scheduler thread_id=AlertNotifier* NOT (alert_actions="summary_index" OR alert_actions="") thread_id="AlertNotifierWorker-2"|stats count(thread_id) as currentval2 ] |where currentval>currentval2 |stats values(currentval) values(currentval2)

Set it to triggered as said above, and let me know if any questions.

[stephanefotso]

I'm sorry it is possible. I did it and it is working.

[vinodmadaan]

Hi Stephan,

Can you please explain how is that possible and how can you get this working?

[juvetm]

i need to know how you did this also

[Runals]

Have your first alert create an event in some index that the second alert looks for.

[NOUMSSI]

Hi,

I think that it's not possible to create an alert that depends on another alert . Splunk alerts are based on reports that run on a regular interval over a set historical time range or in real time (if the report is a real-time search). When the alerts trigger, different actions can take place, such as the sending of an email with the results of the triggering search to a predefined list of people.

This means that you can triggered alert and this action of triggering alert does not means that the alert depends on another.

[vinodmadaan]

Thanks for the answer Noumssi 🙂