Alerting

Alerts Not Firing

merzinger
Splunk Employee
Splunk Employee

Users have been complaining they were not getting email alerts.  While troubleshooting this issue I noticed the alerts were also not being written to the triggered alerts area even though that action is selected in the alert.  I am able to send email notifications using this SPL:  index=_internal | stats count by host
| top 1 host | sendemail to="merzinger@test.com" sendresults=true

To help troubleshoot this some more I created a very simple alert with this SPL:
index=_internal | stats count by host     The search is set to lookup back 15 minutes and the CRON schedule is set for * * * * * to run every minute.  The action for this alert is just to add the event to the Triggered Alerts if results > 0.  This search definitely returns results but the alert actions don't seem to be triggering.  Any help would be appreciated.

Labels (1)
0 Karma

merzinger
Splunk Employee
Splunk Employee

When I run that REST search I see my "MattTest" alert but I don't see it in Activity > Triggered Alerts.  There are no entries in there.

0 Karma

aasabatini
Motivator

Hi @merzinger 

sorry for the late reply, can you tell what you see if you run this search?

index=_audit action=alert_fired ss_app=* | eval ttl=expiration-now() | search ttl>0 | convert ctime(trigger_time) | table trigger_time ss_name severity | rename trigger_time as "Alert Time" ss_name as "Alert Name" severity as "Severity"
“The answer is out there, Neo, and it’s looking for you, and it will find you if you want it to.”
0 Karma

merzinger_prude
Explorer

When I run that search it shows that the alerts haven't fired since early on of 4/22.  They have some KV Store and other issues with this search head cluster so I am going to reform the cluster.  Thanks for your help.

0 Karma

aasabatini
Motivator

Hi @merzinger 

 

If you run this search can you see your triggered alert?

| rest /servicesNS/-/-/saved/searches 
| search title="*" 
| rename dispatch.earliest_time AS "frequency", title AS "title", eai:acl.app AS "app", next_scheduled_time AS "nextRunTime", search AS "query", updated AS "lastUpdated", action.email.to AS "emailTo", action.email.cc AS "emailCC", action.email.subject AS "emailSubject", alert.severity AS "SEV" 
| eval severity=case(SEV == "5", "Critical-5", SEV == "4", "High-4",SEV == "3", "Warning-3",SEV == "2", "Low-2",SEV == "1", "Info-1") 
| table title lastUpdated, nextRunTime, emailTo action.lookup.filename, query, severity
| fillnull value="" 
| sort -lastUpdated
“The answer is out there, Neo, and it’s looking for you, and it will find you if you want it to.”
0 Karma
Get Updates on the Splunk Community!

See just what you’ve been missing | Observability tracks at Splunk University

Looking to sharpen your observability skills so you can better understand how to collect and analyze data from ...

Weezer at .conf25? Say it ain’t so!

Hello Splunkers, The countdown to .conf25 is on-and we've just turned up the volume! We're thrilled to ...

How SC4S Makes Suricata Logs Ingestion Simple

Network security monitoring has become increasingly critical for organizations of all sizes. Splunk has ...