topic Re: How can I enable Splunk email alerts from a Linux server? in Alerting

How can I enable Splunk email alerts from a Linux server?

swdowiarz — Mon, 18 Dec 2017 21:02:55 GMT

I have a problem. I've got Splunk Enterprise installed on Google Cloud Platform on Linux Server and I want to to enable email alerts, but I'm not sure about configuration with SMTP on server. Should I install postfix on a server and provide mail hostname in splunk email settings ? Could anyone help, I would be grateful.

Re: How can I enable Splunk email alerts from a Linux server?

nickhills — Mon, 18 Dec 2017 22:31:24 GMT

Splunk requires a working SMTP server. You can install one on the server, and if your only sending alerts to internal addresses, it should be relativly easy to get your mailserver (or provider) to accept from your Splunk host.

Alternatively, you can configure Splunk to use any SMTP server for which you have credentials - this is probably the better solution, as it will use whatever email system you presently have deployed - and probably less complicated in the long run.

Settings->Server Settings-> Email Settings

Re: How can I enable Splunk email alerts from a Linux server?

swdowiarz — Tue, 19 Dec 2017 08:22:09 GMT

Could you please provide me with more information, I've tried to setup SMTP, as well as I've tried to send email by my email account but in both options it failed. As I know Gooogle Cloud Platfrom is blocking port 25.

Re: How can I enable Splunk email alerts from a Linux server?

nickhills — Tue, 19 Dec 2017 08:47:32 GMT

Without the Splunk server being able to reach something on an SMTP port (TCP25 or TCP587 for TLS), your not going to be able to send any emails.

Have you tried configuring your Splunk server to use the TLS port - If you were using a google/office365 mailserver, Port 25 is normally blocked, but 587 should be fine. As a more general rule, you should always avoid using the insecure ports in favour of the TLS ones.

What mailserver are you configuring, and what settings are you using?

Re: How can I enable Splunk email alerts from a Linux server?

swdowiarz — Tue, 19 Dec 2017 13:31:32 GMT

I've tried to install postfix, as well I was trying to setup splunk to send emails form my gmail account but in both it wasn't working

Re: How can I enable Splunk email alerts from a Linux server?

nickhills — Tue, 19 Dec 2017 13:33:53 GMT

what settings did you use for gmail?

Re: How can I enable Splunk email alerts from a Linux server?

nickhills — Thu, 21 Dec 2017 09:43:51 GMT

How did you get on with this?

Re: How can I enable Splunk email alerts from a Linux server?

swdowiarz — Thu, 21 Dec 2017 10:37:38 GMT

I did it with this tutorial.
https://www.splunk.com/blog/2014/06/27/splunk-alerts-using-gmail-twitter-phone-calls-and-much-more.html

Re: How can I enable Splunk email alerts from a Linux server?

swdowiarz — Thu, 21 Dec 2017 10:38:00 GMT

It still does not work for me 😕

Re: How can I enable Splunk email alerts from a Linux server?

nickhills — Thu, 21 Dec 2017 10:41:29 GMT

Ok, do you see any errors reported if you run this search?
index=_internal sendemail

Re: How can I enable Splunk email alerts from a Linux server?

swdowiarz — Thu, 21 Dec 2017 11:07:25 GMT

12/19/17
8:44:45.363 AM  
12-19-2017 08:44:45.363 +0000 ERROR ScriptRunner - stderr from '/opt/splunk/bin/python /opt/splunk/etc/apps/search/bin/sendemail.py "results_link=http://instance-1:8000/app/search/search?q=%7Cloadjob%20rt_scheduler__admin__search__RMD56cc4d0568864b62f_at_1513672997_1.0%20%7C%20head%201%20%7C%20tail%201&earliest=0&latest=now" "ssname=test alarm" "graceful=True" "trigger_time=1513673084" results_file="/opt/splunk/var/run/splunk/dispatch/rt_scheduler__admin__search__RMD56cc4d0568864b62f_at_1513672997_1.0/per_result_alert/tmp_0.csv.gz"':  ERROR:root:(534, '5.7.14 <https://accounts.google.com/signin/continue?sarp=1&scc=1&plt=AKgnsbvj\n5.7.14 unt3KzFW2DTyz38Sa7SAeySG3Fce0oBpKF0ZfxoisShnmuuZh82ZJEUSbPjqc8dgkWbBcm\n5.7.14 O9OZgjETmRbRvG_jOg4VJtEmFxU1eQgvf2PtSY3GkrU4qK2rl02nGXhTIv2HDdGL0Sx5kz\n5.7.14 3ic761i-XujuqbkGyoWW6emxCvBoMXp8KJQOWlb-tlBv2nOIsIdfiWXt7sscPAwE-g4bIa\n5.7.14 Hvcjr8EisSC7TGuYLeprxiRs56d14> Please log in via your web browser and\n5.7.14 then try again.\n5.7.14  Learn more at\n5.7.14  https://support.google.com/mail/answer/78754 g69sm872707ita.9 - gsmtp') while sending mail to: swdowiarz@groupon.com
host =  instance-1 source = /opt/splunk/var/log/splunk/splunkd.log

12/19/17
8:44:45.362 AM  
2017-12-19 08:44:45,362 +0000 ERROR sendemail:460 - (534, '5.7.14 <https://accounts.google.com/signin/continue?sarp=1&scc=1&plt=AKgnsbvj\n5.7.14 unt3KzFW2DTyz38Sa7SAeySG3Fce0oBpKF0ZfxoisShnmuuZh82ZJEUSbPjqc8dgkWbBcm\n5.7.14 O9OZgjETmRbRvG_jOg4VJtEmFxU1eQgvf2PtSY3GkrU4qK2rl02nGXhTIv2HDdGL0Sx5kz\n5.7.14 3ic761i-XujuqbkGyoWW6emxCvBoMXp8KJQOWlb-tlBv2nOIsIdfiWXt7sscPAwE-g4bIa\n5.7.14 Hvcjr8EisSC7TGuYLeprxiRs56d14> Please log in via your web browser and\n5.7.14 then try again.\n5.7.14  Learn more at\n5.7.14  https://support.google.com/mail/answer/78754 g69sm872707ita.9 - gsmtp') while sending mail to: swdowiarz@groupon.com
host =  instance-1 source = /opt/splunk/var/log/splunk/python.log

12/19/17
8:44:45.361 AM  
2017-12-19 08:44:45,361 +0000 ERROR sendemail:137 - Sending email. subject="Splunk Alert: test alarm", results_link="http://instance-1:8000/app/search/search?q=%7Cloadjob%20rt_scheduler__admin__search__RMD56cc4d0568864b62f_at_1513672997_1.0%20%7C%20head%201%20%7C%20tail%201&earliest=0&latest=now", recipients="[u'swdowiarz@groupon.com']", server="smtp.gmail.com:465"

Re: How can I enable Splunk email alerts from a Linux server?

swdowiarz — Thu, 21 Dec 2017 11:08:26 GMT

those are last errors @nickhillscpl

Re: How can I enable Splunk email alerts from a Linux server?

DavidHourani — Thu, 21 Dec 2017 11:09:15 GMT

Hello swdowiarz,

which port are you using to join the mail host ? Can you please try to run the following from the splunk host to be sure that you can reach that host :

telnet mailHosName portNumber

If that is working please provide an extract from your internal logs for the sendmail command after having used the following command :

yourquerryhere| sendemail to="elvis@splunk.com" sendresults=true

Docs here: https://docs.splunk.com/Documentation/SplunkCloud/6.6.3/SearchReference/Sendemail

Regards,
David

Re: How can I enable Splunk email alerts from a Linux server?

nickhills — Thu, 21 Dec 2017 11:11:06 GMT

Do you have 2 factor authentication on your account?
If so you will need to generate and use an app-specific-password.

Did you look at the google link specified in the error:
https://support.google.com/mail/answer/78754