Why is my alert not triggering?

[ambuj]

Would anyone know why is my alert not triggering?

I have created a simple scheduled alert which should send an email if result count > 0. When I use the "Open in Search" menu then I can see some results. I also see this message on the alert page: There are no fired events for this alert.

The setup looks like this:

[boromir]

I have a similar issue..... my Real-Time alerts are triggering always, no problem with that, mails are being received, so no issue there. However the scheduled alerts are not always triggering. I have days when the alerts are fine, and then i have days when though the search finds an event, the alert is never triggered. We do suspect that the difference between system time and the time stamp of the event might have something into that, but will let you know if we manage to prove it.

regards!

[Nisha18789]

Hi @boromir , since you mentioned your environment has realtime searches and they run on time, there is a possibility that some of your normal alerts might be getting skipped.

Please check if the alert which is missing schedule sometimes is getting skipped using below query:

index=_internal sourcetype=scheduler status="skipped"  savedsearch_name="<name of the alert which is not firing always>"

Also, to get a bigger picture if your environment is facing skipped searches issue, you can run below search :

index=_internal sourcetype=scheduler (status="success" OR status="skipped" OR status="continued")
| top status

Let me know what you find out by this test.

[isoutamo]

Another reason could be (as you are already thinking) that events are late and missed alerts time slot. You could check this by looking event creation (_time) vs indexing time (_indextime). If this is greater than your schedule period then this is the reason. It is easily fixed by adding earliest=X _index_earliest=Y where Xis greater than your max delay and Y is your previous schedule.

r. Ismo

[templets]

I had the same problem. My search, which generated results, but never triggered, ended with:

| table Time host CPU_Load CPU_limit email_to cc_to

When I changed this to explicitly add a fields statement:

| table Time host CPU_Load CPU_limit email_to cc_to
| fields Time host CPU_Load CPU_limit email_to cc_to

I miraculously started getting alerts.

[L1_marrera]

@templets This one worked for me. The alert was working on Verbose but not on Fast nor Smart mode. I used "|fields *" at the beginning of my search and it started working on the other modes too.

[abhijitsaoji]

Hi, I am facing same issue, my real time alert is not working at all. It is neither appearing in the triggered alerts nor sending any emails. I have changed the alert type to scheduled - every hour on 30th minute and worked like a charm. Not sure what would be the issue with real time, I have read few comments about latency and ran the query supplied but latency is coming in seconds so probably it can be ruled out. any other thoughts, please let me know.

[nick405060]

You should ignore the "There are no fired events for this alert" message. I have the same message if I click on any of my alerts, and they are all sending email alerts out fine.

The first thing you should do is to Edit Actions and add "Add to Triggered Alerts." Then go to Activity (in the top right corner of Splunk) and select Triggered Alerts and monitor that page. If the alert triggers there, which I'm guessing it will, then you know it's an email problem. If that's the case, there are already a bunch of Splunk Answers that address email alerting problems, including one by me back when I was having the same problem:

https://answers.splunk.com/answers/681118/why-are-the-email-alerts-not-being-sent-anymore.html

[nick405060]

Also, I believe if you add "Add to Triggered Alerts" that will fix the "There are no fired events for this alert" message. (I don't mind that error message and only add to Triggered Alerts for temporary debugging purposes)

[rajanshrivastav]

Check your email settings, try to send emails manually from cmd if that is working then check in your app setting where you've defined mail settings.

On the server side, mail setting should be in only one place, may be you've defined it some where else as well.

Thanks,
Rajan Shrivastav