How can I enable Splunk email alerts from a Linux server?

[swdowiarz]

Hi

I have a problem. I've got Splunk Enterprise installed on Google Cloud Platform on Linux Server and I want to to enable email alerts, but I'm not sure about configuration with SMTP on server. Should I install postfix on a server and provide mail hostname in splunk email settings ? Could anyone help, I would be grateful.

[DavidHourani]

Hello swdowiarz,

which port are you using to join the mail host ? Can you please try to run the following from the splunk host to be sure that you can reach that host :

telnet mailHosName portNumber

If that is working please provide an extract from your internal logs for the sendmail command after having used the following command :

yourquerryhere| sendemail to="elvis@splunk.com" sendresults=true

Docs here: https://docs.splunk.com/Documentation/SplunkCloud/6.6.3/SearchReference/Sendemail

Regards,
David

[nickhills]

Splunk requires a working SMTP server. You can install one on the server, and if your only sending alerts to internal addresses, it should be relativly easy to get your mailserver (or provider) to accept from your Splunk host.

Alternatively, you can configure Splunk to use any SMTP server for which you have credentials - this is probably the better solution, as it will use whatever email system you presently have deployed - and probably less complicated in the long run.

Settings->Server Settings-> Email Settings

[swdowiarz]

Could you please provide me with more information, I've tried to setup SMTP, as well as I've tried to send email by my email account but in both options it failed. As I know Gooogle Cloud Platfrom is blocking port 25.

[nickhills]

Without the Splunk server being able to reach something on an SMTP port (TCP25 or TCP587 for TLS), your not going to be able to send any emails.

Have you tried configuring your Splunk server to use the TLS port - If you were using a google/office365 mailserver, Port 25 is normally blocked, but 587 should be fine. As a more general rule, you should always avoid using the insecure ports in favour of the TLS ones.

What mailserver are you configuring, and what settings are you using?

[swdowiarz]

I've tried to install postfix, as well I was trying to setup splunk to send emails form my gmail account but in both it wasn't working

[nickhills]

what settings did you use for gmail?

[swdowiarz]

I did it with this tutorial.
https://www.splunk.com/blog/2014/06/27/splunk-alerts-using-gmail-twitter-phone-calls-and-much-more.h...

[nickhills]

Ok, do you see any errors reported if you run this search?
index=_internal sendemail

[swdowiarz]

12/19/17
8:44:45.363 AM  
12-19-2017 08:44:45.363 +0000 ERROR ScriptRunner - stderr from '/opt/splunk/bin/python /opt/splunk/etc/apps/search/bin/sendemail.py "results_link=http://instance-1:8000/app/search/search?q=%7Cloadjob%20rt_scheduler__admin__search__RMD56cc4d0568864b62f_at_1513672997_1.0%20%7C%20head%201%20%7C%20tail%201&earliest=0&latest=now" "ssname=test alarm" "graceful=True" "trigger_time=1513673084" results_file="/opt/splunk/var/run/splunk/dispatch/rt_scheduler__admin__search__RMD56cc4d0568864b62f_at_1513672997_1.0/per_result_alert/tmp_0.csv.gz"':  ERROR:root:(534, '5.7.14 <https://accounts.google.com/signin/continue?sarp=1&scc=1&plt=AKgnsbvj\n5.7.14 unt3KzFW2DTyz38Sa7SAeySG3Fce0oBpKF0ZfxoisShnmuuZh82ZJEUSbPjqc8dgkWbBcm\n5.7.14 O9OZgjETmRbRvG_jOg4VJtEmFxU1eQgvf2PtSY3GkrU4qK2rl02nGXhTIv2HDdGL0Sx5kz\n5.7.14 3ic761i-XujuqbkGyoWW6emxCvBoMXp8KJQOWlb-tlBv2nOIsIdfiWXt7sscPAwE-g4bIa\n5.7.14 Hvcjr8EisSC7TGuYLeprxiRs56d14> Please log in via your web browser and\n5.7.14 then try again.\n5.7.14  Learn more at\n5.7.14  https://support.google.com/mail/answer/78754 g69sm872707ita.9 - gsmtp') while sending mail to: swdowiarz@groupon.com
host =  instance-1 source = /opt/splunk/var/log/splunk/splunkd.log

12/19/17
8:44:45.362 AM  
2017-12-19 08:44:45,362 +0000 ERROR sendemail:460 - (534, '5.7.14 <https://accounts.google.com/signin/continue?sarp=1&scc=1&plt=AKgnsbvj\n5.7.14 unt3KzFW2DTyz38Sa7SAeySG3Fce0oBpKF0ZfxoisShnmuuZh82ZJEUSbPjqc8dgkWbBcm\n5.7.14 O9OZgjETmRbRvG_jOg4VJtEmFxU1eQgvf2PtSY3GkrU4qK2rl02nGXhTIv2HDdGL0Sx5kz\n5.7.14 3ic761i-XujuqbkGyoWW6emxCvBoMXp8KJQOWlb-tlBv2nOIsIdfiWXt7sscPAwE-g4bIa\n5.7.14 Hvcjr8EisSC7TGuYLeprxiRs56d14> Please log in via your web browser and\n5.7.14 then try again.\n5.7.14  Learn more at\n5.7.14  https://support.google.com/mail/answer/78754 g69sm872707ita.9 - gsmtp') while sending mail to: swdowiarz@groupon.com
host =  instance-1 source = /opt/splunk/var/log/splunk/python.log

12/19/17
8:44:45.361 AM  
2017-12-19 08:44:45,361 +0000 ERROR sendemail:137 - Sending email. subject="Splunk Alert: test alarm", results_link="http://instance-1:8000/app/search/search?q=%7Cloadjob%20rt_scheduler__admin__search__RMD56cc4d0568864b62f_at_1513672997_1.0%20%7C%20head%201%20%7C%20tail%201&earliest=0&latest=now", recipients="[u'swdowiarz@groupon.com']", server="smtp.gmail.com:465"

[swdowiarz]

those are last errors @nickhillscpl

[nickhills]

Do you have 2 factor authentication on your account?
If so you will need to generate and use an app-specific-password.

Did you look at the google link specified in the error:
https://support.google.com/mail/answer/78754

[nickhills]

How did you get on with this?

[swdowiarz]

It still does not work for me 😕