Alerting
Highlighted

Why triggered real-time alert does not send an email when condition is met?

Builder

Hi,
I have problems understanding a situation. First, the problem manifested itself when a colleague approached me with the issue that his schedule real-time search is not sending emails when a certain event is happening in the log file. I couldn't really comprehend why, as the alert was created and is listed in Triggered Alerts. The condition was "Always" and the alert mode "Once per result", so I don't see a reason why the email isn't being sent.

I have verified that the search head is sending other alerts, so there is no issue in the connectivity to the smtp server.

Secondly, I tried cloning this search, changed it from real-time to "-1d to now". I'm not getting emails, but I am seing the alerts in the "Triggered Alerts". I don't really understand this combination of behaviour. Either it shouldn't be in "Triggered Alerts" and not send an email, or it should be listed AND it should send an email.

Or am I missing something?

Highlighted

Re: Why triggered real-time alert does not send an email when condition is met?

Engager

Check to make sure your Splunk instances as well as the system that you are collecting logs from are synced to NTP.

Having system time off on any of these can absolutely screw up alerting.

That includes validating that the timezones are correct.

0 Karma
Highlighted

Re: Why triggered real-time alert does not send an email when condition is met?

Builder

Yes, we are using NTP for all servers involved and the timezone is the same for all.

0 Karma
Highlighted

Re: Why triggered real-time alert does not send an email when condition is met?

SplunkTrust
SplunkTrust

did you check for any errors in:

index=_internal ( sourcetype=scheduler alert_actions="email" ) OR ( sourcetype=splunk_python "sendemail" )
Highlighted

Re: Why triggered real-time alert does not send an email when condition is met?

Builder

Yes. No indication of problems: status=success for 100% of events returned.

0 Karma
Highlighted

Re: Why triggered real-time alert does not send an email when condition is met?

SplunkTrust
SplunkTrust

You did check the basic stuff, like sending email possible at all ;)? Maybe someone did change something outside your Splunk setup?

0 Karma
Highlighted

Re: Why triggered real-time alert does not send an email when condition is met?

Builder

Again: yes. Another alert email is being sent and I have checked the connection to the mail server. I would also expect any problems with email sending to end up in the internal logs of Splunk.

0 Karma
Highlighted

Re: Why triggered real-time alert does not send an email when condition is met?

Contributor

can you just give an attempt by
using default values in email settings.

http://docs.splunk.com/Documentation/Splunk/6.1.3/Alert/Setupalertactions

0 Karma
Highlighted

Re: Why triggered real-time alert does not send an email when condition is met?

Builder

That's pretty much what I'm doing, actually.

0 Karma