All Apps and Add-ons

Need help with alerting mechanism

theouhuios
Motivator

Hello

I am setting up alerts and facing few issues in this. I have a custom condition to be met for an alert to be triggered and this search runs every 5 mins. So if my custom condition is I>B where I is the count of number of incidents for a Workgroup and B is the threshold for that Workgroup it needs to send an email. And I set the throttling at 20 mins.

Now the issue is I need to make the alert come in a way that if an alert has been created for one WorkGroup then it shouldn't send out an alert for that workgroup till the data goes below the threshold. I am doing this because when an alert is received user will be going through the WG and he doesn't need another alert for the same WG again unless the WG's issue has been solved where it goes below threshold (I<B) and again it crosses the threshold. At the same time the search needs to generate alerts for other Workgroups which have crossed thresholds but follow the same mechanism as above.

Any ideas?

Regards

theou

Tags (1)
0 Karma

Ledion_Bitincka
Splunk Employee
Splunk Employee

Here's some more info on how to get the lookup method working, assume that we're working with the internal logs and we want to alert for a sourcetype when the high threshold is above 100 and want to throttle until the count drops below 10

  1. define a file based lookup called throttle (that contains three fields: sourcetype,curr,prev)
  2. define your search as follows (remove the comments)
    //1. report search that that generates stats of interests
    index=_internal 
    | stats count AS curr BY sourcetype 

    //2. load and update the lookup "state" table 
    | append [inputlookup throttle | rename curr AS prev] 
    | stats first(curr) AS curr, first(prev) AS prev BY sourcetype 
    | outputlookup throttle 

    //3. apply your condition search 
    | search curr>100 (prev="" OR prev<10)     

When this search completes it will only output the results that you need need to alert for - so it is best to set your alert condition to "number of results" "greater than" 0

0 Karma

Ledion_Bitincka
Splunk Employee
Splunk Employee

I'd recommend that you start with an empty file - you need the header in order to define the lookup via the manager.

0 Karma

theouhuios
Motivator

I will try this out. In the 1 statement do we need to put the stats in a file throttle ? I am not familiar with how append works, hence the question. will go through the documentation for append.

0 Karma

Ledion_Bitincka
Splunk Employee
Splunk Employee

One way to solve this is through keeping state - ie you're not looking for time base throttling but for alert state based throttling. To achieve this you have to somehow keep state for each workgroup, there are two ways to do this:

  1. use a lookup table to maintain previous and current alert state for each workgroup and then trigger the alert only if current > high threshold and previous < low threshold
  2. build a custom search command that maintains state and encapsulates your alerting logic
0 Karma

theouhuios
Motivator

Can you please elaborate on the first option. I am kind of lost on how will that impact.

0 Karma
*NEW* Splunk Love Promo!
Snag a $25 Visa Gift Card for Giving Your Review!

It's another Splunk Love Special! For a limited time, you can review one of our select Splunk products through Gartner Peer Insights and receive a $25 Visa gift card!

Review:





Or Learn More in Our Blog >>