Splunk Search

Is it possible to create an alert that depends on another alert to be triggered?

vinodmadaan
Path Finder

Hi Guys,

I am asking this question out of curiosity (don't even know if this is possible!).

The question is: Is it possible to create an alert that depends on another alert? For example, if Alert-1 is triggered then only Alert-2 will be triggered (point to note is that : both use different searches, so the option that both can have same search is out)?

Thanks in advance,
Vinod.

0 Karma
1 Solution

stephanefotso
Motivator

Tere is an app wich can help you manage your alerts as you need: http://challengepost.com/software/alert-manager
Concerning my own researches here is what i was able to do.

First, notice that the _internal index let you manage your alerts. For example, you can get your alert thread_id which can be AlertNotifierWorker-0 AlertNotifierWorker-1,.......................
Each alert has a unique AlertNotifierWorker, which is increment each time the alert is triggered.
Now let suppose that you have two alerts.
Alert1 with thread_id=AlertNotifierWorker-0 and currentcount0 as the current count of alerts triggered,
Alert2 with thread_id=AlertNotifierWorker-2 and currentcount2 as the current count of alerts triggered.
suppositions:
1. We have schedulded Alert1 to triggered at 5 min window
2. We have scheduled Alert2 to triggered at 1 min window, only if Alert1 is triggered.
3. the started time is 6 am.
4. Alert1 is always triggered

Conditions:
Alert1 is triggered when the word "error" is seen for the last 5min in your events.
Alert2 if Alert1 is triggered, means, currentcount0>currentcount2.

initialisations:
if you

Algorithm

  1. At 6H:5min am, Alert1 is triggered, AlertNotifierWorker-0 is increment, means AlertNotifierWorker-0==1
  2. At 6H:6min am, Alert2 is triggered, because AlertNotifierWorker-0>AlertNotifierWorker-2. AlertNotifierWorker-2 is then increment, means AlertNotifierWorker-2==1.
  3. From 6H:6min To 6H: 9min, Since AlertNotifierWorker-0==AlertNotifierWorker-2, no alert is triggered.
  4. At 6H:10min, Alert1 is triggered, means AlertNotifierWorker-0==2
  5. At 6H:11min am, Alert2 is triggered, because AlertNotifierWorker-0>AlertNotifierWorker-2. AlertNotifierWorker-2 is then increment, means AlertNotifierWorker-2==2.
  6. From 6H:6min To 6H: 9min, Since AlertNotifierWorker-0==AlertNotifierWorker-2, no alert is triggered.
  7. .........................
  8. ...............................and so on

implementation

Alert1:
search query: index=* OR index=_* "error"
Launch the search and save it as an alert, and set it as described above

Alert2:
Search qury:

index=_internal sourcetype=scheduler thread_id=AlertNotifier* NOT (alert_actions="summary_index" OR alert_actions="") thread_id="AlertNotifierWorker-0"|stats count(thread_id) as currentval|join [search index=_internal sourcetype=scheduler thread_id=AlertNotifier* NOT (alert_actions="summary_index" OR alert_actions="") thread_id="AlertNotifierWorker-2"|stats count(thread_id) as currentval2 ] |where currentval>currentval2 |stats values(currentval) values(currentval2)

Set it to triggered as said above, and let me know if any questions.

SGF

View solution in original post

stephanefotso
Motivator

Tere is an app wich can help you manage your alerts as you need: http://challengepost.com/software/alert-manager
Concerning my own researches here is what i was able to do.

First, notice that the _internal index let you manage your alerts. For example, you can get your alert thread_id which can be AlertNotifierWorker-0 AlertNotifierWorker-1,.......................
Each alert has a unique AlertNotifierWorker, which is increment each time the alert is triggered.
Now let suppose that you have two alerts.
Alert1 with thread_id=AlertNotifierWorker-0 and currentcount0 as the current count of alerts triggered,
Alert2 with thread_id=AlertNotifierWorker-2 and currentcount2 as the current count of alerts triggered.
suppositions:
1. We have schedulded Alert1 to triggered at 5 min window
2. We have scheduled Alert2 to triggered at 1 min window, only if Alert1 is triggered.
3. the started time is 6 am.
4. Alert1 is always triggered

Conditions:
Alert1 is triggered when the word "error" is seen for the last 5min in your events.
Alert2 if Alert1 is triggered, means, currentcount0>currentcount2.

initialisations:
if you

Algorithm

  1. At 6H:5min am, Alert1 is triggered, AlertNotifierWorker-0 is increment, means AlertNotifierWorker-0==1
  2. At 6H:6min am, Alert2 is triggered, because AlertNotifierWorker-0>AlertNotifierWorker-2. AlertNotifierWorker-2 is then increment, means AlertNotifierWorker-2==1.
  3. From 6H:6min To 6H: 9min, Since AlertNotifierWorker-0==AlertNotifierWorker-2, no alert is triggered.
  4. At 6H:10min, Alert1 is triggered, means AlertNotifierWorker-0==2
  5. At 6H:11min am, Alert2 is triggered, because AlertNotifierWorker-0>AlertNotifierWorker-2. AlertNotifierWorker-2 is then increment, means AlertNotifierWorker-2==2.
  6. From 6H:6min To 6H: 9min, Since AlertNotifierWorker-0==AlertNotifierWorker-2, no alert is triggered.
  7. .........................
  8. ...............................and so on

implementation

Alert1:
search query: index=* OR index=_* "error"
Launch the search and save it as an alert, and set it as described above

Alert2:
Search qury:

index=_internal sourcetype=scheduler thread_id=AlertNotifier* NOT (alert_actions="summary_index" OR alert_actions="") thread_id="AlertNotifierWorker-0"|stats count(thread_id) as currentval|join [search index=_internal sourcetype=scheduler thread_id=AlertNotifier* NOT (alert_actions="summary_index" OR alert_actions="") thread_id="AlertNotifierWorker-2"|stats count(thread_id) as currentval2 ] |where currentval>currentval2 |stats values(currentval) values(currentval2)

Set it to triggered as said above, and let me know if any questions.

SGF

stephanefotso
Motivator

I'm sorry it is possible. I did it and it is working.

SGF
0 Karma

vinodmadaan
Path Finder

Hi Stephan,

Can you please explain how is that possible and how can you get this working?

0 Karma

juvetm
Communicator

i need to know how you did this also

0 Karma

Runals
Motivator

Have your first alert create an event in some index that the second alert looks for.

0 Karma

NOUMSSI
Builder

Hi,

I think that it's not possible to create an alert that depends on another alert . Splunk alerts are based on reports that run on a regular interval over a set historical time range or in real time (if the report is a real-time search). When the alerts trigger, different actions can take place, such as the sending of an email with the results of the triggering search to a predefined list of people.

This means that you can triggered alert and this action of triggering alert does not means that the alert depends on another.

vinodmadaan
Path Finder

Thanks for the answer Noumssi 🙂

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...