I copied alert_actions.conf [email] stanza from .../default to .../local directory as alert_actions.conf
I set up an alert to fire on the hour + 30. It does this successfully, however the email does not get sent.
When I created the alert the mailserver = was successfully changed in the alert_actions.conf
To make sure that the email could be sent, I successfully did a telnet 25 and sent an email from splunk@ to the recipient.
I noticed that after the alert fires, this entry changed in the alert_actions.conf file, even though I opted for no authentication:
auth_password = $1$0A==
2014-03-27 11:30:03,019 Central Daylight Time ERROR sendemail:357 - Sending email. subject="Splunk Alert: Load Avg > 2",
(this worked from a browser)
recipients="['my valid email address']"
Solved. The problem was in the documentation stating that the sender could be just 'splunk' and the system would add the domain. Not so, and there are two places where you have to define the sender and it needs to be a full splunk@