Alerting

How can I enable Splunk email alerts from a Linux server?

Path Finder

Hi

I have a problem. I've got Splunk Enterprise installed on Google Cloud Platform on Linux Server and I want to to enable email alerts, but I'm not sure about configuration with SMTP on server. Should I install postfix on a server and provide mail hostname in splunk email settings ? Could anyone help, I would be grateful.

0 Karma

Super Champion

Hello swdowiarz,

which port are you using to join the mail host ? Can you please try to run the following from the splunk host to be sure that you can reach that host :

telnet mailHosName portNumber

If that is working please provide an extract from your internal logs for the sendmail command after having used the following command :

yourquerryhere| sendemail to="elvis@splunk.com" sendresults=true

Docs here: https://docs.splunk.com/Documentation/SplunkCloud/6.6.3/SearchReference/Sendemail

Regards,
David

0 Karma

Ultra Champion

Splunk requires a working SMTP server. You can install one on the server, and if your only sending alerts to internal addresses, it should be relativly easy to get your mailserver (or provider) to accept from your Splunk host.

Alternatively, you can configure Splunk to use any SMTP server for which you have credentials - this is probably the better solution, as it will use whatever email system you presently have deployed - and probably less complicated in the long run.

Settings->Server Settings-> Email Settings

0 Karma

Path Finder

Could you please provide me with more information, I've tried to setup SMTP, as well as I've tried to send email by my email account but in both options it failed. As I know Gooogle Cloud Platfrom is blocking port 25.

0 Karma

Ultra Champion

Without the Splunk server being able to reach something on an SMTP port (TCP25 or TCP587 for TLS), your not going to be able to send any emails.

Have you tried configuring your Splunk server to use the TLS port - If you were using a google/office365 mailserver, Port 25 is normally blocked, but 587 should be fine. As a more general rule, you should always avoid using the insecure ports in favour of the TLS ones.

What mailserver are you configuring, and what settings are you using?

0 Karma

Path Finder

I've tried to install postfix, as well I was trying to setup splunk to send emails form my gmail account but in both it wasn't working

0 Karma

Ultra Champion

what settings did you use for gmail?

0 Karma

Path Finder
0 Karma

Ultra Champion

Ok, do you see any errors reported if you run this search?
index=_internal sendemail

0 Karma

Path Finder
12/19/17
8:44:45.363 AM  
12-19-2017 08:44:45.363 +0000 ERROR ScriptRunner - stderr from '/opt/splunk/bin/python /opt/splunk/etc/apps/search/bin/sendemail.py "results_link=http://instance-1:8000/app/search/search?q=%7Cloadjob%20rt_scheduler__admin__search__RMD56cc4d0568864b62f_at_1513672997_1.0%20%7C%20head%201%20%7C%20tail%201&earliest=0&latest=now" "ssname=test alarm" "graceful=True" "trigger_time=1513673084" results_file="/opt/splunk/var/run/splunk/dispatch/rt_scheduler__admin__search__RMD56cc4d0568864b62f_at_1513672997_1.0/per_result_alert/tmp_0.csv.gz"':  ERROR:root:(534, '5.7.14 <https://accounts.google.com/signin/continue?sarp=1&scc=1&plt=AKgnsbvj\n5.7.14 unt3KzFW2DTyz38Sa7SAeySG3Fce0oBpKF0ZfxoisShnmuuZh82ZJEUSbPjqc8dgkWbBcm\n5.7.14 O9OZgjETmRbRvG_jOg4VJtEmFxU1eQgvf2PtSY3GkrU4qK2rl02nGXhTIv2HDdGL0Sx5kz\n5.7.14 3ic761i-XujuqbkGyoWW6emxCvBoMXp8KJQOWlb-tlBv2nOIsIdfiWXt7sscPAwE-g4bIa\n5.7.14 Hvcjr8EisSC7TGuYLeprxiRs56d14> Please log in via your web browser and\n5.7.14 then try again.\n5.7.14  Learn more at\n5.7.14  https://support.google.com/mail/answer/78754 g69sm872707ita.9 - gsmtp') while sending mail to: swdowiarz@groupon.com
host =  instance-1 source = /opt/splunk/var/log/splunk/splunkd.log

12/19/17
8:44:45.362 AM  
2017-12-19 08:44:45,362 +0000 ERROR sendemail:460 - (534, '5.7.14 <https://accounts.google.com/signin/continue?sarp=1&scc=1&plt=AKgnsbvj\n5.7.14 unt3KzFW2DTyz38Sa7SAeySG3Fce0oBpKF0ZfxoisShnmuuZh82ZJEUSbPjqc8dgkWbBcm\n5.7.14 O9OZgjETmRbRvG_jOg4VJtEmFxU1eQgvf2PtSY3GkrU4qK2rl02nGXhTIv2HDdGL0Sx5kz\n5.7.14 3ic761i-XujuqbkGyoWW6emxCvBoMXp8KJQOWlb-tlBv2nOIsIdfiWXt7sscPAwE-g4bIa\n5.7.14 Hvcjr8EisSC7TGuYLeprxiRs56d14> Please log in via your web browser and\n5.7.14 then try again.\n5.7.14  Learn more at\n5.7.14  https://support.google.com/mail/answer/78754 g69sm872707ita.9 - gsmtp') while sending mail to: swdowiarz@groupon.com
host =  instance-1 source = /opt/splunk/var/log/splunk/python.log

12/19/17
8:44:45.361 AM  
2017-12-19 08:44:45,361 +0000 ERROR sendemail:137 - Sending email. subject="Splunk Alert: test alarm", results_link="http://instance-1:8000/app/search/search?q=%7Cloadjob%20rt_scheduler__admin__search__RMD56cc4d0568864b62f_at_1513672997_1.0%20%7C%20head%201%20%7C%20tail%201&earliest=0&latest=now", recipients="[u'swdowiarz@groupon.com']", server="smtp.gmail.com:465"
0 Karma

Path Finder

those are last errors @nickhillscpl

0 Karma

Ultra Champion

Do you have 2 factor authentication on your account?
If so you will need to generate and use an app-specific-password.

Did you look at the google link specified in the error:
https://support.google.com/mail/answer/78754

0 Karma

Ultra Champion

How did you get on with this?

0 Karma

Path Finder

It still does not work for me 😕

0 Karma
State of Splunk Careers

Access the Splunk Careers Report to see real data that shows how Splunk mastery increases your value and job satisfaction.

Find out what your skills are worth!