Alerting

Why triggered real-time alert does not send an email when condition is met?

Builder

Hi,
I have problems understanding a situation. First, the problem manifested itself when a colleague approached me with the issue that his schedule real-time search is not sending emails when a certain event is happening in the log file. I couldn't really comprehend why, as the alert was created and is listed in Triggered Alerts. The condition was "Always" and the alert mode "Once per result", so I don't see a reason why the email isn't being sent.

I have verified that the search head is sending other alerts, so there is no issue in the connectivity to the smtp server.

Secondly, I tried cloning this search, changed it from real-time to "-1d to now". I'm not getting emails, but I am seing the alerts in the "Triggered Alerts". I don't really understand this combination of behaviour. Either it shouldn't be in "Triggered Alerts" and not send an email, or it should be listed AND it should send an email.

Or am I missing something?

Contributor

can you just give an attempt by
using default values in email settings.

http://docs.splunk.com/Documentation/Splunk/6.1.3/Alert/Setupalertactions

0 Karma

Builder

That's pretty much what I'm doing, actually.

0 Karma

Engager

Check to make sure your Splunk instances as well as the system that you are collecting logs from are synced to NTP.

Having system time off on any of these can absolutely screw up alerting.

That includes validating that the timezones are correct.

0 Karma

Builder

Yes, we are using NTP for all servers involved and the timezone is the same for all.

0 Karma

SplunkTrust
SplunkTrust

did you check for any errors in:

index=_internal ( sourcetype=scheduler alert_actions="email" ) OR ( sourcetype=splunk_python "sendemail" )

Builder

Yes. No indication of problems: status=success for 100% of events returned.

0 Karma

SplunkTrust
SplunkTrust

You did check the basic stuff, like sending email possible at all ;)? Maybe someone did change something outside your Splunk setup?

0 Karma

Builder

Again: yes. Another alert email is being sent and I have checked the connection to the mail server. I would also expect any problems with email sending to end up in the internal logs of Splunk.

0 Karma