Alerting

How to send alert based on number of occurrences by using email trigger

subramanianp789
New Member

Hi,
I have an requirement that need to schedule the below search query for every 2 mins(it can be given in corn schedule */2 * * * *) but it should not trigger mail immediately even breach threshold and the after specified time limit which will be mentioned in the lookup csv file, consider field as "count_threshold"=3

Now, after 6 mins (ie 2 mins * 3 = 6 mins) the email has to trigger if the search query breach the threshold(or if the results greater than 0)

Below is the existing search:

| eval "Alert Status" = case((' Virtual Bytes'<=manual_threshold3),"NORMAL", (' Virtual Bytes'>manual_threshold4),"CRITICAL", (' Virtual Bytes'>manual_threshold3 AND ' Virtual Bytes'<=manual_threshold4),"WARNING")
| search "Alert Status"="CRITICAL"
| table  Host," PID","Process Name"," Virtual Bytes","Alert Status"

How to achieve this? And how to add the condition to send mail after 6 mins.

Thanks.

0 Karma

woodcock
Esteemed Legend

The way to do this is to write out the results of this search to a lookup file with ... | outputlookup. Then create another scheduled search that runs on a different cron schedule that mines the outputs of the first search using | inputlookup ... and applies your throttling/email logic. This second search (or even a third search) can trim/cleanup the lookup so that it doesn't grow out of control in size.

0 Karma

subramanianp789
New Member

Thanks a lot for your valuable comments. Can you please explain more in detail about the inputlookup. It would be great if you provide me some sample steps to proceed further. Thanks again.

Also, how to pass the value from lookup csv to cron schedule or trigger condition, where to include. Please share your thoughts on this

0 Karma

woodcock
Esteemed Legend

My answer gives all the basic pieces; how you put them together is up to you. The main thing is to put as much of your thresholding and throttling logic inside of the search SPL as possible (try to keep the alert action number of results and is greater than 0).

0 Karma
Get Updates on the Splunk Community!

Introduction to Splunk Observability Cloud - Building a Resilient Hybrid Cloud

Introduction to Splunk Observability Cloud - Building a Resilient Hybrid Cloud  In today’s fast-paced digital ...

Observability protocols to know about

Observability protocols define the specifications or formats for collecting, encoding, transporting, and ...

Take Your Breath Away with Splunk Risk-Based Alerting (RBA)

WATCH NOW!The Splunk Guide to Risk-Based Alerting is here to empower your SOC like never before. Join Haylee ...