I'm trying to get our Splunk to integrate with Amazon SES but I'm not have had any luck so far. E-mails can be sent out on the Linux box using sendmail so there is nothing wrong with Authentication and the e-mail delivery part. Somehow Splunk refuses to send the e-mail and fails on Authentication failed. This is the error I'm seeing on /usr/local/splunk/var/log/splunkd.log
01-10-2014 19:31:10.319 -0500 ERROR ScriptRunner - stderr from '/usr/local/splunk/etc/apps/search/bin/sendemail.py': ERROR:root:(535, 'Authentication Credentials Invalid') while sending mail to: firstname.lastname@example.org
I think that using a AWS static credential isn't the best way to do this, actually using the "ec2 role" would be the best and most secure way, does anybody have a version os this "sendemail.py" changed to use AWS SDK/API to use credentials from the EC2 instance?
Thank you for getting back. I tried these settings. I hoped the auth details would be incorrect but it isn't. I'm going to work with Splunk support on this one after we purchase the product and update this thread if we find a fix. For now, I'm switching to the local MTA. Thanks again.
Well then, you'll have to ensure you have set up the appropriate TLS (assuming you are using that) port, SMTP username, and SMTP password for your SES configuration. This is all done in "email alert settings." Probably the host will be something like email-smtp.us-east-1.amazonaws.com:465 or email-smtp.us-east-1.amazonaws.com:587. If various combinations of things there don't work I'd start looking at running sendmail command manually to troubleshoot, as the link above mentions, and/or checking logs on the SES side to see if there's any clue - maybe as simple as the wrong SMTP user/pass.
jbrodsky_splunk - We are trying to move away from the local MTA and instead direct the application to directly connect to SES. All our apps including Bugzilla does this and wanted Splunk to do so too.
That depends on what you have under "mail host" in email settings. If your local linux host can send out just fine then maybe use its local MTA - set the "mail host" to localhost and see where that gets you. Also, see this: http://answers.splunk.com/answers/3225/saved-searches-not-emailing-out
There is nothing on /var/log/maillog. A follow up question. Would Splunk while sending e-mails externally (like in this case SES) log to /var/log/maillog. I thought it would only log to splunkd.log as /var/log/maillog is used by MTA on the system. Correct me if I'm wrong here.