Alerting

How can I enable Splunk email alerts from a Linux server?

swdowiarz
Path Finder

Hi

I have a problem. I've got Splunk Enterprise installed on Google Cloud Platform on Linux Server and I want to to enable email alerts, but I'm not sure about configuration with SMTP on server. Should I install postfix on a server and provide mail hostname in splunk email settings ? Could anyone help, I would be grateful.

0 Karma

DavidHourani
Super Champion

Hello swdowiarz,

which port are you using to join the mail host ? Can you please try to run the following from the splunk host to be sure that you can reach that host :

telnet mailHosName portNumber

If that is working please provide an extract from your internal logs for the sendmail command after having used the following command :

yourquerryhere| sendemail to="elvis@splunk.com" sendresults=true

Docs here: https://docs.splunk.com/Documentation/SplunkCloud/6.6.3/SearchReference/Sendemail

Regards,
David

0 Karma

nickhills
Ultra Champion

Splunk requires a working SMTP server. You can install one on the server, and if your only sending alerts to internal addresses, it should be relativly easy to get your mailserver (or provider) to accept from your Splunk host.

Alternatively, you can configure Splunk to use any SMTP server for which you have credentials - this is probably the better solution, as it will use whatever email system you presently have deployed - and probably less complicated in the long run.

Settings->Server Settings-> Email Settings

If my comment helps, please give it a thumbs up!
0 Karma

swdowiarz
Path Finder

Could you please provide me with more information, I've tried to setup SMTP, as well as I've tried to send email by my email account but in both options it failed. As I know Gooogle Cloud Platfrom is blocking port 25.

0 Karma

nickhills
Ultra Champion

Without the Splunk server being able to reach something on an SMTP port (TCP25 or TCP587 for TLS), your not going to be able to send any emails.

Have you tried configuring your Splunk server to use the TLS port - If you were using a google/office365 mailserver, Port 25 is normally blocked, but 587 should be fine. As a more general rule, you should always avoid using the insecure ports in favour of the TLS ones.

What mailserver are you configuring, and what settings are you using?

If my comment helps, please give it a thumbs up!
0 Karma

swdowiarz
Path Finder

I've tried to install postfix, as well I was trying to setup splunk to send emails form my gmail account but in both it wasn't working

0 Karma

nickhills
Ultra Champion

what settings did you use for gmail?

If my comment helps, please give it a thumbs up!
0 Karma

swdowiarz
Path Finder
0 Karma

nickhills
Ultra Champion

Ok, do you see any errors reported if you run this search?
index=_internal sendemail

If my comment helps, please give it a thumbs up!
0 Karma

swdowiarz
Path Finder
12/19/17
8:44:45.363 AM  
12-19-2017 08:44:45.363 +0000 ERROR ScriptRunner - stderr from '/opt/splunk/bin/python /opt/splunk/etc/apps/search/bin/sendemail.py "results_link=http://instance-1:8000/app/search/search?q=%7Cloadjob%20rt_scheduler__admin__search__RMD56cc4d0568864b62f_at_1513672997_1.0%20%7C%20head%201%20%7C%20tail%201&earliest=0&latest=now" "ssname=test alarm" "graceful=True" "trigger_time=1513673084" results_file="/opt/splunk/var/run/splunk/dispatch/rt_scheduler__admin__search__RMD56cc4d0568864b62f_at_1513672997_1.0/per_result_alert/tmp_0.csv.gz"':  ERROR:root:(534, '5.7.14 <https://accounts.google.com/signin/continue?sarp=1&scc=1&plt=AKgnsbvj\n5.7.14 unt3KzFW2DTyz38Sa7SAeySG3Fce0oBpKF0ZfxoisShnmuuZh82ZJEUSbPjqc8dgkWbBcm\n5.7.14 O9OZgjETmRbRvG_jOg4VJtEmFxU1eQgvf2PtSY3GkrU4qK2rl02nGXhTIv2HDdGL0Sx5kz\n5.7.14 3ic761i-XujuqbkGyoWW6emxCvBoMXp8KJQOWlb-tlBv2nOIsIdfiWXt7sscPAwE-g4bIa\n5.7.14 Hvcjr8EisSC7TGuYLeprxiRs56d14> Please log in via your web browser and\n5.7.14 then try again.\n5.7.14  Learn more at\n5.7.14  https://support.google.com/mail/answer/78754 g69sm872707ita.9 - gsmtp') while sending mail to: swdowiarz@groupon.com
host =  instance-1 source = /opt/splunk/var/log/splunk/splunkd.log

12/19/17
8:44:45.362 AM  
2017-12-19 08:44:45,362 +0000 ERROR sendemail:460 - (534, '5.7.14 <https://accounts.google.com/signin/continue?sarp=1&scc=1&plt=AKgnsbvj\n5.7.14 unt3KzFW2DTyz38Sa7SAeySG3Fce0oBpKF0ZfxoisShnmuuZh82ZJEUSbPjqc8dgkWbBcm\n5.7.14 O9OZgjETmRbRvG_jOg4VJtEmFxU1eQgvf2PtSY3GkrU4qK2rl02nGXhTIv2HDdGL0Sx5kz\n5.7.14 3ic761i-XujuqbkGyoWW6emxCvBoMXp8KJQOWlb-tlBv2nOIsIdfiWXt7sscPAwE-g4bIa\n5.7.14 Hvcjr8EisSC7TGuYLeprxiRs56d14> Please log in via your web browser and\n5.7.14 then try again.\n5.7.14  Learn more at\n5.7.14  https://support.google.com/mail/answer/78754 g69sm872707ita.9 - gsmtp') while sending mail to: swdowiarz@groupon.com
host =  instance-1 source = /opt/splunk/var/log/splunk/python.log

12/19/17
8:44:45.361 AM  
2017-12-19 08:44:45,361 +0000 ERROR sendemail:137 - Sending email. subject="Splunk Alert: test alarm", results_link="http://instance-1:8000/app/search/search?q=%7Cloadjob%20rt_scheduler__admin__search__RMD56cc4d0568864b62f_at_1513672997_1.0%20%7C%20head%201%20%7C%20tail%201&earliest=0&latest=now", recipients="[u'swdowiarz@groupon.com']", server="smtp.gmail.com:465"
0 Karma

swdowiarz
Path Finder

those are last errors @nickhillscpl

0 Karma

nickhills
Ultra Champion

Do you have 2 factor authentication on your account?
If so you will need to generate and use an app-specific-password.

Did you look at the google link specified in the error:
https://support.google.com/mail/answer/78754

If my comment helps, please give it a thumbs up!
0 Karma

nickhills
Ultra Champion

How did you get on with this?

If my comment helps, please give it a thumbs up!
0 Karma

swdowiarz
Path Finder

It still does not work for me 😕

0 Karma
Register for .conf21 Now! Go Vegas or Go Virtual!

How will you .conf21? You decide! Go in-person in Las Vegas, 10/18-10/21, or go online with .conf21 Virtual, 10/19-10/20.