Alerting

Email alerts with logics

tsunamii
Path Finder

For example, I want to accomplish this with only one alert saved search: to send email notification to admin_a if the search returns between 10 and 100 events within 5 mins (or when the search is ran), but to run some additional scripts or send email alert to admin_a and manager, if more than 100 events returned within 5 minutes.

Tags (1)
0 Karma
1 Solution

martin_mueller
SplunkTrust
SplunkTrust

There are several things mixed up here, so I'll try to untangle.

First I'll assume you have a search that returns some count field in an event, and you want to alert if count >= 10. To do that, you can either add | where count >= 10 to your search and alert on "search returns an event", or add the where count >= 10 as the alert condition.
If you actually have the events returned from your search, you can instead set the alert condition to "alert if number of events is greater than" and enter your number.

Second, you want to email different people depending on the count. Set your email recipient to $result.recipient$ and add this to your search: ... | eval recipient = if (count < 100, "[email protected]", "[email protected],[email protected]")
If you're actually returning events, add this instead: ... | eventstats count | eval if(...)

Third, you want to run a script based on the count, but don't want a second alert. Run the script every time, and let the script abort if the count is below the threshold. If you can't modify your script then create a simple wrapper script to make this decision.

View solution in original post

splunkIT
Splunk Employee
Splunk Employee

This appears to work on splunk 6.3:

"error" | stats count | eval recipient=case(count > 3500, "[email protected]", count >= 500, "[email protected]", 1==1, null()) | where isnotnull(recipient)

AND in the to field you specify:
$result.recipient$


So this says,

if we have > 3500 records then
send to recipient1
else if you have > 500 records
then send to recipient 2
else
do not send any mail (sets recipient to null)

0 Karma

splunkIT
Splunk Employee
Splunk Employee
0 Karma

martin_mueller
SplunkTrust
SplunkTrust

There are several things mixed up here, so I'll try to untangle.

First I'll assume you have a search that returns some count field in an event, and you want to alert if count >= 10. To do that, you can either add | where count >= 10 to your search and alert on "search returns an event", or add the where count >= 10 as the alert condition.
If you actually have the events returned from your search, you can instead set the alert condition to "alert if number of events is greater than" and enter your number.

Second, you want to email different people depending on the count. Set your email recipient to $result.recipient$ and add this to your search: ... | eval recipient = if (count < 100, "[email protected]", "[email protected],[email protected]")
If you're actually returning events, add this instead: ... | eventstats count | eval if(...)

Third, you want to run a script based on the count, but don't want a second alert. Run the script every time, and let the script abort if the count is below the threshold. If you can't modify your script then create a simple wrapper script to make this decision.

Got questions? Get answers!

Join the Splunk Community Slack to learn, troubleshoot, and make connections with fellow Splunk practitioners in real time!

Meet up IRL or virtually!

Join Splunk User Groups to connect and learn in-person by region or remotely by topic or industry.

Get Updates on the Splunk Community!

Get Agentic with Splunk Lantern: Connect to Cisco Cloud Control, Transform ...

Splunk Lantern is Splunk’s customer success center that provides practical guidance from Splunk experts on key ...

July Community Events: Master ITSI 5.0 & Automate Splunk

Struggling with alert fatigue or feeling like you're spending more time on infrastructure maintenance than ...

New Release of Federated Search: Bringing Splunk Analytics to More of Your Data

Organizations today are generating more data than ever and storing it across cloud object stores, data lakes, ...