Integrating Splunk with Amazon SES to send e-mails

[kkossery]

I'm trying to get our Splunk to integrate with Amazon SES but I'm not have had any luck so far. E-mails can be sent out on the Linux box using sendmail so there is nothing wrong with Authentication and the e-mail delivery part. Somehow Splunk refuses to send the e-mail and fails on Authentication failed. This is the error I'm seeing on /usr/local/splunk/var/log/splunkd.log

01-10-2014 19:31:10.319 -0500 ERROR ScriptRunner - stderr from '/usr/local/splunk/etc/apps/search/bin/sendemail.py': ERROR:root:(535, 'Authentication Credentials Invalid') while sending mail to: email@email.com

[freaklin]

I think that using a AWS static credential isn't the best way to do this, actually using the "ec2 role" would be the best and most secure way, does anybody have a version os this "sendemail.py" changed to use AWS SDK/API to use credentials from the EC2 instance?

[kkossery]

The issue was due to populating IAM credentials instead of using SES credentials. This is very easy to overlook and a bit confusing on the AWS side.
Thanks for everybody's help!

[jbrodsky_splunk]

OK - good luck and sorry it wasn't simple. Do update this when support weighs in.

[kkossery]

Thank you for getting back. I tried these settings. I hoped the auth details would be incorrect but it isn't. I'm going to work with Splunk support on this one after we purchase the product and update this thread if we find a fix. For now, I'm switching to the local MTA. Thanks again.

[jbrodsky_splunk]

And also, this: http://answers.splunk.com/answers/27220/how-to-send-splunk-email-alerts-through-aws-ses. Not the initial response, but the later one that talks about how to use the SMTP service.

[jbrodsky_splunk]

Well then, you'll have to ensure you have set up the appropriate TLS (assuming you are using that) port, SMTP username, and SMTP password for your SES configuration. This is all done in "email alert settings." Probably the host will be something like email-smtp.us-east-1.amazonaws.com:465 or email-smtp.us-east-1.amazonaws.com:587. If various combinations of things there don't work I'd start looking at running sendmail command manually to troubleshoot, as the link above mentions, and/or checking logs on the SES side to see if there's any clue - maybe as simple as the wrong SMTP user/pass.

[kkossery]

jbrodsky_splunk - We are trying to move away from the local MTA and instead direct the application to directly connect to SES. All our apps including Bugzilla does this and wanted Splunk to do so too.

[jbrodsky_splunk]

That depends on what you have under "mail host" in email settings. If your local linux host can send out just fine then maybe use its local MTA - set the "mail host" to localhost and see where that gets you. Also, see this: http://answers.splunk.com/answers/3225/saved-searches-not-emailing-out

[kkossery]

There is nothing on /var/log/maillog. A follow up question. Would Splunk while sending e-mails externally (like in this case SES) log to /var/log/maillog. I thought it would only log to splunkd.log as /var/log/maillog is used by MTA on the system. Correct me if I'm wrong here.

[jbrodsky_splunk]

Any clues in your /var/log/maillog?