Users have been complaining they were not getting email alerts. While troubleshooting this issue I noticed the alerts were also not being written to the triggered alerts area even though that action is selected in the alert. I am able to send email notifications using this SPL: index=_internal | stats count by host
| top 1 host | sendemail to="merzinger@test.com" sendresults=true
To help troubleshoot this some more I created a very simple alert with this SPL:
index=_internal | stats count by host The search is set to lookup back 15 minutes and the CRON schedule is set for * * * * * to run every minute. The action for this alert is just to add the event to the Triggered Alerts if results > 0. This search definitely returns results but the alert actions don't seem to be triggering. Any help would be appreciated.
When I run that REST search I see my "MattTest" alert but I don't see it in Activity > Triggered Alerts. There are no entries in there.
Hi @merzinger
sorry for the late reply, can you tell what you see if you run this search?
index=_audit action=alert_fired ss_app=* | eval ttl=expiration-now() | search ttl>0 | convert ctime(trigger_time) | table trigger_time ss_name severity | rename trigger_time as "Alert Time" ss_name as "Alert Name" severity as "Severity"
When I run that search it shows that the alerts haven't fired since early on of 4/22. They have some KV Store and other issues with this search head cluster so I am going to reform the cluster. Thanks for your help.
Hi @merzinger
If you run this search can you see your triggered alert?
| rest /servicesNS/-/-/saved/searches
| search title="*"
| rename dispatch.earliest_time AS "frequency", title AS "title", eai:acl.app AS "app", next_scheduled_time AS "nextRunTime", search AS "query", updated AS "lastUpdated", action.email.to AS "emailTo", action.email.cc AS "emailCC", action.email.subject AS "emailSubject", alert.severity AS "SEV"
| eval severity=case(SEV == "5", "Critical-5", SEV == "4", "High-4",SEV == "3", "Warning-3",SEV == "2", "Low-2",SEV == "1", "Info-1")
| table title lastUpdated, nextRunTime, emailTo action.lookup.filename, query, severity
| fillnull value=""
| sort -lastUpdated