Splunk Search

Is it possible to create an alert that depends on another alert to be triggered?

vinodmadaan
Path Finder

Hi Guys,

I am asking this question out of curiosity (don't even know if this is possible!).

The question is: Is it possible to create an alert that depends on another alert? For example, if Alert-1 is triggered then only Alert-2 will be triggered (point to note is that : both use different searches, so the option that both can have same search is out)?

Thanks in advance,
Vinod.

0 Karma
1 Solution

stephanefotso
Motivator

Tere is an app wich can help you manage your alerts as you need: http://challengepost.com/software/alert-manager
Concerning my own researches here is what i was able to do.

First, notice that the _internal index let you manage your alerts. For example, you can get your alert thread_id which can be AlertNotifierWorker-0 AlertNotifierWorker-1,.......................
Each alert has a unique AlertNotifierWorker, which is increment each time the alert is triggered.
Now let suppose that you have two alerts.
Alert1 with thread_id=AlertNotifierWorker-0 and currentcount0 as the current count of alerts triggered,
Alert2 with thread_id=AlertNotifierWorker-2 and currentcount2 as the current count of alerts triggered.
suppositions:
1. We have schedulded Alert1 to triggered at 5 min window
2. We have scheduled Alert2 to triggered at 1 min window, only if Alert1 is triggered.
3. the started time is 6 am.
4. Alert1 is always triggered

Conditions:
Alert1 is triggered when the word "error" is seen for the last 5min in your events.
Alert2 if Alert1 is triggered, means, currentcount0>currentcount2.

initialisations:
if you

Algorithm

  1. At 6H:5min am, Alert1 is triggered, AlertNotifierWorker-0 is increment, means AlertNotifierWorker-0==1
  2. At 6H:6min am, Alert2 is triggered, because AlertNotifierWorker-0>AlertNotifierWorker-2. AlertNotifierWorker-2 is then increment, means AlertNotifierWorker-2==1.
  3. From 6H:6min To 6H: 9min, Since AlertNotifierWorker-0==AlertNotifierWorker-2, no alert is triggered.
  4. At 6H:10min, Alert1 is triggered, means AlertNotifierWorker-0==2
  5. At 6H:11min am, Alert2 is triggered, because AlertNotifierWorker-0>AlertNotifierWorker-2. AlertNotifierWorker-2 is then increment, means AlertNotifierWorker-2==2.
  6. From 6H:6min To 6H: 9min, Since AlertNotifierWorker-0==AlertNotifierWorker-2, no alert is triggered.
  7. .........................
  8. ...............................and so on

implementation

Alert1:
search query: index=* OR index=_* "error"
Launch the search and save it as an alert, and set it as described above

Alert2:
Search qury:

index=_internal sourcetype=scheduler thread_id=AlertNotifier* NOT (alert_actions="summary_index" OR alert_actions="") thread_id="AlertNotifierWorker-0"|stats count(thread_id) as currentval|join [search index=_internal sourcetype=scheduler thread_id=AlertNotifier* NOT (alert_actions="summary_index" OR alert_actions="") thread_id="AlertNotifierWorker-2"|stats count(thread_id) as currentval2 ] |where currentval>currentval2 |stats values(currentval) values(currentval2)

Set it to triggered as said above, and let me know if any questions.

View solution in original post

stephanefotso
Motivator

Tere is an app wich can help you manage your alerts as you need: http://challengepost.com/software/alert-manager
Concerning my own researches here is what i was able to do.

First, notice that the _internal index let you manage your alerts. For example, you can get your alert thread_id which can be AlertNotifierWorker-0 AlertNotifierWorker-1,.......................
Each alert has a unique AlertNotifierWorker, which is increment each time the alert is triggered.
Now let suppose that you have two alerts.
Alert1 with thread_id=AlertNotifierWorker-0 and currentcount0 as the current count of alerts triggered,
Alert2 with thread_id=AlertNotifierWorker-2 and currentcount2 as the current count of alerts triggered.
suppositions:
1. We have schedulded Alert1 to triggered at 5 min window
2. We have scheduled Alert2 to triggered at 1 min window, only if Alert1 is triggered.
3. the started time is 6 am.
4. Alert1 is always triggered

Conditions:
Alert1 is triggered when the word "error" is seen for the last 5min in your events.
Alert2 if Alert1 is triggered, means, currentcount0>currentcount2.

initialisations:
if you

Algorithm

  1. At 6H:5min am, Alert1 is triggered, AlertNotifierWorker-0 is increment, means AlertNotifierWorker-0==1
  2. At 6H:6min am, Alert2 is triggered, because AlertNotifierWorker-0>AlertNotifierWorker-2. AlertNotifierWorker-2 is then increment, means AlertNotifierWorker-2==1.
  3. From 6H:6min To 6H: 9min, Since AlertNotifierWorker-0==AlertNotifierWorker-2, no alert is triggered.
  4. At 6H:10min, Alert1 is triggered, means AlertNotifierWorker-0==2
  5. At 6H:11min am, Alert2 is triggered, because AlertNotifierWorker-0>AlertNotifierWorker-2. AlertNotifierWorker-2 is then increment, means AlertNotifierWorker-2==2.
  6. From 6H:6min To 6H: 9min, Since AlertNotifierWorker-0==AlertNotifierWorker-2, no alert is triggered.
  7. .........................
  8. ...............................and so on

implementation

Alert1:
search query: index=* OR index=_* "error"
Launch the search and save it as an alert, and set it as described above

Alert2:
Search qury:

index=_internal sourcetype=scheduler thread_id=AlertNotifier* NOT (alert_actions="summary_index" OR alert_actions="") thread_id="AlertNotifierWorker-0"|stats count(thread_id) as currentval|join [search index=_internal sourcetype=scheduler thread_id=AlertNotifier* NOT (alert_actions="summary_index" OR alert_actions="") thread_id="AlertNotifierWorker-2"|stats count(thread_id) as currentval2 ] |where currentval>currentval2 |stats values(currentval) values(currentval2)

Set it to triggered as said above, and let me know if any questions.

stephanefotso
Motivator

I'm sorry it is possible. I did it and it is working.

0 Karma

vinodmadaan
Path Finder

Hi Stephan,

Can you please explain how is that possible and how can you get this working?

0 Karma

juvetm
Communicator

i need to know how you did this also

0 Karma

Runals
Motivator

Have your first alert create an event in some index that the second alert looks for.

0 Karma

NOUMSSI
Builder

Hi,

I think that it's not possible to create an alert that depends on another alert . Splunk alerts are based on reports that run on a regular interval over a set historical time range or in real time (if the report is a real-time search). When the alerts trigger, different actions can take place, such as the sending of an email with the results of the triggering search to a predefined list of people.

This means that you can triggered alert and this action of triggering alert does not means that the alert depends on another.

vinodmadaan
Path Finder

Thanks for the answer Noumssi 🙂

0 Karma
Get Updates on the Splunk Community!

Improve Your Security Posture

Watch NowImprove Your Security PostureCustomers are at the center of everything we do at Splunk and security ...

Maximize the Value from Microsoft Defender with Splunk

 Watch NowJoin Splunk and Sens Consulting for this Security Edition Tech TalkWho should attend:  Security ...

This Week's Community Digest - Splunk Community Happenings [6.27.22]

Get the latest news and updates from the Splunk Community here! News From Splunk Answers ✍️ Splunk Answers is ...