Why is Email Alert not triggering though there is search result

[shradha14]

Hi,

I have created an email alert with cron schedule of every 4 hours, though I can see that even if there is search result, randomly email triggering is not happening.

Also, I made sure to use simpler splunk commands which will be a bit faster in terms of execution.

Can someone please suggest what could be the reason in such skipping of an email.

[shradha14]

Its a project requirement that we have multiple dashboards/searches.
As per customer requirement we have to get this email notification no matter what every 4 hours. 

Is there any solution that though query is in queued state, alert output will get triggered?

[richgalloway]

Have you looked at the scheduler log as suggested to see why the query did not run?  Only after knowing the reason for the failure can you hope to correct it.

[richgalloway]

Is it a problem of the alert not triggering or the email not getting delivered?  You can check the "Triggered Alerts" page for the former and index=_internal for the latter.

[isoutamo]

Have you checked that those alerts are fired and email has sent by splunk? Just use internal indexes to see that. One example how to look those https://community.splunk.com/t5/Alerting/How-to-troubleshoot-why-I-m-not-getting-email-alerts-from-S...

r. Ismo

[shradha14]

I have tested email alert as well before set up.  It triggered with the cron schedule correctly. I have observed sometimes at the time of cron schedule, alert output has the output "Waiting for queued jobs".

Is this the reason email is not triggering ??  Each time I have to re-schedule cron to 5 mins and manually run it.

Can someone please suggest what can be done even if query is in queued state and still it has to trigger an email ?

[richgalloway]

There should be something in the scheduler log (index=_internal source=*scheduler.log*) explaining why the alert didn't run.

It sounds like you have too many searches trying to run at the same time so some have to wait (queue) for resources to become available.  Consider rescheduling or disabling some searches.

[isoutamo]

Another place to check is MC’s Search-> Scheduler -> individual node or something. Look skipped and deferred searches to see how well your scheduler is working.