topic Re: Alerts Not Firing in Alerting

Alerts Not Firing

merzinger — Wed, 21 Apr 2021 15:47:33 GMT

Users have been complaining they were not getting email alerts. While troubleshooting this issue I noticed the alerts were also not being written to the triggered alerts area even though that action is selected in the alert. I am able to send email notifications using this SPL: index=_internal | stats count by host
| top 1 host | sendemail to="merzinger@test.com" sendresults=true

To help troubleshoot this some more I created a very simple alert with this SPL:
index=_internal | stats count by host The search is set to lookup back 15 minutes and the CRON schedule is set for * * * * * to run every minute. The action for this alert is just to add the event to the Triggered Alerts if results > 0. This search definitely returns results but the alert actions don't seem to be triggering. Any help would be appreciated.

Re: Alerts Not Firing

aasabatini — Wed, 21 Apr 2021 15:53:07 GMT

Hi @merzinger

If you run this search can you see your triggered alert?

| rest /servicesNS/-/-/saved/searches | search title="*" | rename dispatch.earliest_time AS "frequency", title AS "title", eai:acl.app AS "app", next_scheduled_time AS "nextRunTime", search AS "query", updated AS "lastUpdated", action.email.to AS "emailTo", action.email.cc AS "emailCC", action.email.subject AS "emailSubject", alert.severity AS "SEV" | eval severity=case(SEV == "5", "Critical-5", SEV == "4", "High-4",SEV == "3", "Warning-3",SEV == "2", "Low-2",SEV == "1", "Info-1") | table title lastUpdated, nextRunTime, emailTo action.lookup.filename, query, severity | fillnull value="" | sort -lastUpdated

Re: Alerts Not Firing

merzinger — Wed, 21 Apr 2021 16:24:44 GMT

When I run that REST search I see my "MattTest" alert but I don't see it in Activity > Triggered Alerts. There are no entries in there.

Re: Alerts Not Firing

aasabatini — Fri, 23 Apr 2021 09:59:24 GMT

Hi @merzinger

sorry for the late reply, can you tell what you see if you run this search?

Re: Alerts Not Firing

merzinger_prude — Fri, 23 Apr 2021 21:52:59 GMT

When I run that search it shows that the alerts haven't fired since early on of 4/22. They have some KV Store and other issues with this search head cluster so I am going to reform the cluster. Thanks for your help.