Reporting

Integrating Splunk with Amazon SES to send e-mails

kkossery
Communicator

I'm trying to get our Splunk to integrate with Amazon SES but I'm not have had any luck so far. E-mails can be sent out on the Linux box using sendmail so there is nothing wrong with Authentication and the e-mail delivery part. Somehow Splunk refuses to send the e-mail and fails on Authentication failed. This is the error I'm seeing on /usr/local/splunk/var/log/splunkd.log

01-10-2014 19:31:10.319 -0500 ERROR ScriptRunner - stderr from '/usr/local/splunk/etc/apps/search/bin/sendemail.py': ERROR:root:(535, 'Authentication Credentials Invalid') while sending mail to: email@email.com

Tags (3)
0 Karma

freaklin
Path Finder

I think that using a AWS static credential isn't the best way to do this, actually using the "ec2 role" would be the best and most secure way, does anybody have a version os this "sendemail.py" changed to use AWS SDK/API to use credentials from the EC2 instance?

0 Karma

kkossery
Communicator

The issue was due to populating IAM credentials instead of using SES credentials. This is very easy to overlook and a bit confusing on the AWS side.
Thanks for everybody's help!

0 Karma

jbrodsky_splunk
Splunk Employee
Splunk Employee

OK - good luck and sorry it wasn't simple. Do update this when support weighs in.

0 Karma

kkossery
Communicator

Thank you for getting back. I tried these settings. I hoped the auth details would be incorrect but it isn't. I'm going to work with Splunk support on this one after we purchase the product and update this thread if we find a fix. For now, I'm switching to the local MTA. Thanks again.

0 Karma

jbrodsky_splunk
Splunk Employee
Splunk Employee

And also, this: http://answers.splunk.com/answers/27220/how-to-send-splunk-email-alerts-through-aws-ses. Not the initial response, but the later one that talks about how to use the SMTP service.

0 Karma

jbrodsky_splunk
Splunk Employee
Splunk Employee

Well then, you'll have to ensure you have set up the appropriate TLS (assuming you are using that) port, SMTP username, and SMTP password for your SES configuration. This is all done in "email alert settings." Probably the host will be something like email-smtp.us-east-1.amazonaws.com:465 or email-smtp.us-east-1.amazonaws.com:587. If various combinations of things there don't work I'd start looking at running sendmail command manually to troubleshoot, as the link above mentions, and/or checking logs on the SES side to see if there's any clue - maybe as simple as the wrong SMTP user/pass.

kkossery
Communicator

jbrodsky_splunk - We are trying to move away from the local MTA and instead direct the application to directly connect to SES. All our apps including Bugzilla does this and wanted Splunk to do so too.

0 Karma

jbrodsky_splunk
Splunk Employee
Splunk Employee

That depends on what you have under "mail host" in email settings. If your local linux host can send out just fine then maybe use its local MTA - set the "mail host" to localhost and see where that gets you. Also, see this: http://answers.splunk.com/answers/3225/saved-searches-not-emailing-out

0 Karma

kkossery
Communicator

There is nothing on /var/log/maillog. A follow up question. Would Splunk while sending e-mails externally (like in this case SES) log to /var/log/maillog. I thought it would only log to splunkd.log as /var/log/maillog is used by MTA on the system. Correct me if I'm wrong here.

0 Karma

jbrodsky_splunk
Splunk Employee
Splunk Employee

Any clues in your /var/log/maillog?

0 Karma
Take the 2021 Splunk Career Survey

Help us learn about how Splunk has
impacted your career by taking the 2021 Splunk Career Survey.

Earn $50 in Amazon cash!