Alerting

How to send alert based on number of occurrences by using email trigger

subramanianp789
New Member

Hi,
I have an requirement that need to schedule the below search query for every 2 mins(it can be given in corn schedule */2 * * * *) but it should not trigger mail immediately even breach threshold and the after specified time limit which will be mentioned in the lookup csv file, consider field as "count_threshold"=3

Now, after 6 mins (ie 2 mins * 3 = 6 mins) the email has to trigger if the search query breach the threshold(or if the results greater than 0)

Below is the existing search:

| eval "Alert Status" = case((' Virtual Bytes'<=manual_threshold3),"NORMAL", (' Virtual Bytes'>manual_threshold4),"CRITICAL", (' Virtual Bytes'>manual_threshold3 AND ' Virtual Bytes'<=manual_threshold4),"WARNING")
| search "Alert Status"="CRITICAL"
| table  Host," PID","Process Name"," Virtual Bytes","Alert Status"

How to achieve this? And how to add the condition to send mail after 6 mins.

Thanks.

0 Karma

woodcock
Esteemed Legend

The way to do this is to write out the results of this search to a lookup file with ... | outputlookup. Then create another scheduled search that runs on a different cron schedule that mines the outputs of the first search using | inputlookup ... and applies your throttling/email logic. This second search (or even a third search) can trim/cleanup the lookup so that it doesn't grow out of control in size.

0 Karma

subramanianp789
New Member

Thanks a lot for your valuable comments. Can you please explain more in detail about the inputlookup. It would be great if you provide me some sample steps to proceed further. Thanks again.

Also, how to pass the value from lookup csv to cron schedule or trigger condition, where to include. Please share your thoughts on this

0 Karma

woodcock
Esteemed Legend

My answer gives all the basic pieces; how you put them together is up to you. The main thing is to put as much of your thresholding and throttling logic inside of the search SPL as possible (try to keep the alert action number of results and is greater than 0).

0 Karma
Career Survey
First 500 qualified respondents will receive a $20 gift card! Tell us about your professional Splunk journey.
Get Updates on the Splunk Community!

Tech Talk Recap | Mastering Threat Hunting

Mastering Threat HuntingDive into the world of threat hunting, exploring the key differences between ...

Observability for AI Applications: Troubleshooting Latency

If you’re working with proprietary company data, you’re probably going to have a locally hosted LLM or many ...

Splunk AI Assistant for SPL vs. ChatGPT: Which One is Better?

In the age of AI, every tool promises to make our lives easier. From summarizing content to writing code, ...