Alerting

How to troubleshoot why I'm not getting email alerts from Splunk?

brent_weaver
Builder

I have set up email on my Search Head. I am able to send a test email message using the following:

whatever search | sendemail to="me@mydomain.com"

But when I set up alerts, I do not get an email, and I am sure there are results. What am I missing? I cannot find any log files that indicate a problem. Any suggestions are more than welcome! Thanks!

0 Karma

woodcock
Esteemed Legend

Check your spam folders.

0 Karma

hortonew
Builder

So Splunk has a couple log files that you might want to look through first to see if you notice any problems. The process here would be saved search runs, should log an entry in the scheduler log, then finally the python.log should contain an entry for sendemail.py for your given search.

This will show you ever time your search executed. If the "fired" field is > 0, you know it matched on an event. If there are instances where it matched, then move on to the next step.

index=_internal source=*scheduler.log host=<> savedsearch_name=<>

This will show Splunk is calling sendemail.py, which should indicate at least an attempt to send out the email.

index=_internal source=*python.log host=<>

You could further limit the search based on the "results_link" field. If you see entries here for your saved search, this means Splunk called the python script that should have sent out an email

If you don't see anything in the second ,it's possible you missed a step in your alerts. If you can, screenshot them and link to them here for further troubleshooting.

0 Karma

brent_weaver
Builder

I guess I needed to have python installed, which I did. Now I get this message when I run the sendemail.py script:

[root@cdopeusvmesem01 ~]# python /opt/splunk/etc/apps/search/bin/sendemail.py
Traceback (most recent call last):
  File "/opt/splunk/etc/apps/search/bin/sendemail.py", line 1, in 
    import re, time, splunk.Intersplunk, splunk.mining.dcutils as dcu
ImportError: No module named splunk.Intersplunk

Is there a problem with the sendemail.py script?

0 Karma

hortonew
Builder

Python comes with Splunk, which contains it's own environment variables. In order to run sendemail.py like splunk would, you can call:

splunk cmb python /opt/splunk/etc/apps/search/bin/sendemail.py
0 Karma

MuS
Legend

No need to install Python, since this will be provided by Splunk.... try this:

   /opt/splunk/bin/splunk cmd python

This will start Splunk's Python which is used by sendmail.py

I assume you have some limitation of either sending out emails form this server or some mismatch in the mail server setting on your Splunk. Run this search from the UI to see your current setting:

 | rest /services/configs/conf-alert_actions | table mailserver

or the CLI:

 /opt/splunk/bin/splunk search "| rest /services/configs/conf-alert_actions | table mailserver"

cheers, MuS

brent_weaver
Builder

This is still not working. I am able to send mail from the linux server itself. In spunk I am getting the following message still:

12/6/15 
10:03:38.586 AM 
2015-12-06 10:03:38,586 -0500 ERROR sendemail:355 - Connection unexpectedly closed while sending mail to: bver@rege.com
host = cdopeusvmesem01 sourcetype = splunk_python
12/6/15 
10:03:38.586 AM 
2015-12-06 10:03:38,586 -0500 ERROR sendemail:113 - Sending email. subject="Splunk Alert: TestAlerts", results_link="https://cdopesem01:8000/app/search/@go?sid=scheduler__admin__search__TestAlerts_at_1449412200_76116", recipients="[u'amil@domain.com']", server="mail.pcloud.com"
host = cdopeusvmesem01 sourcetype = splunk_python
12/6/15 
10:00:09.945 AM 
2015-12-06 10:00:09,945 -0500 ERROR sendemail:355 - Connection unexpectedly closed while sending mail to: sleppers@gmail.com
host = cdopeuem01 sourcetype = splunk_python
12/6/15 
10:00:09.945 AM 
2015-12-06 10:00:09,945 -0500 ERROR sendemail:113 - Sending email. subject="Splunk Alert: Brents New Test", results_link="https://cdopesem01:8000/app/search/@go?sid=scheduler__admin__search__RMD5114f93919c0bc2ab_at_1449414000_76273", recipients="[u'bj@gmail.com']", server="mail.pcloud.com"

The above is from the log file. What am I missing here?!?!?! It seems that I should not have to configure sendmail at all in linux?!? Any help is much appreciated as I have been working on this for a while. Please note that I edited the above to exclude my email and server info.

0 Karma

brent_weaver
Builder

Thank you SO MUCH! This got me what I needed to see. The following was in the log python log file:

sendemail:355 - Connection unexpectedly closed while sending mail to:

The following was in the scheduler log file:

12-05-2015 17:15:05.960 +0000 INFO  SavedSplunker - savedsearch_id="212040597;search;Brents Test Alert", user="212040597", app="search", savedsearch_name="Brents Test Alert", status=success, digest_mode=1, scheduled_time=1449335700, dispatch_time=1449335703, run_time=2.146, result_count=20, alert_actions="email", sid="scheduler__212040597__search__RMD5aab130161880edb9_at_1449335700_69055", suppressed=0, thread_id="AlertNotifierWorker-0"

I did not have sendmail installed on this server . Do I need to install and configure it? This is so confusing because when I run a search and pipe it thru sendemail to= I get the messages. Is there a how-to for setting up mail in splunk. What is the meaning of life? 🙂

Thanks everyone for your help.

0 Karma

hortonew
Builder

sendemail.py exists in $SPLUNK_HOME/etc/apps/search/bin and comes with Splunk by default so you should have it. Are you configuring your alert to send an email in the alert actions, or are you trying to pipe to sendemail in your saved search itself?

0 Karma

brent_weaver
Builder

I am trying to send email from alerts and it is failied. Piping to sendemail works just fine. I dont get it!

0 Karma

brent_weaver
Builder

Ignore that I found it!

0 Karma
Get Updates on the Splunk Community!

Detecting Remote Code Executions With the Splunk Threat Research Team

WATCH NOWRemote code execution (RCE) vulnerabilities pose a significant risk to organizations. If exploited, ...

Enter the Splunk Community Dashboard Challenge for Your Chance to Win!

The Splunk Community Dashboard Challenge is underway! This is your chance to showcase your skills in creating ...

.conf24 | Session Scheduler is Live!!

.conf24 is happening June 11 - 14 in Las Vegas, and we are thrilled to announce that the conference catalog ...