In my Splunk 6.3.3 search head cluster, why is an alert email not being sent to a distribution email list?

[jdunlea]

I have a scheduled search that finds results successfully.

However, the search will NOT email the results as part of an alert action when the "to" field is set to a distribution email list. EG:
"security_employees@mycompany.com".

It works just fine when I set the "to" field to an actual user's email address. EG: "john.doe@mycompany.com".

What is even more strange, is that when I run the SAME search in the search app and simply append the "sendemail" command to the end and I set the "to" field to the original distribution list "security_employees@mycompany.com" then it DOES work. Example below:

index=ABC sourcetype=123 "find this event" | stats count by host | sendemail to="security_employees@mycompany.com" subject="Email Alert"

Anybody have any ideas here?

NOTE: We are running 6.3.3 in a Search Head Cluster environment.

[jwelch_splunk]

Email results from a saved search are being sent from the mailserver setting in alert_action.conf
email
mailserver = [:]

When you use a | sendemail command you are inferring send the email from LOCALHOST unless you over ride it with |sendemail server=xxxx

So what this might imply is as follows.

1. when your email comes from your local host, it is getting different rules applied than when it is sent direct to your server in alert_actions.conf.

Check the email header of the |sendemail command

Check the email header of the savedsearch results

I think you will find that there are two different paths being used and I bet this is your issue.

One path is taking you out to your MX record most likely and sending to DL's is rejected because they are not known as vailid addresses because your org does not want external sources hitting DL's

2nd path is staying internal using your mail routers and therefore it is not rejecting email to your DL's

Okie
John

[harsmarvania57]

Have you tried to send email from command line from each server which are running in SHC environment? I guess you are using linux. So command is something like

[username@server1 ~]$ mailx -s "hi" youremail@yourcompany.com
.
EOT
Null message body; hope that's ok
[username@server1 ~]$

If above command works find and you got email in your inbox then next troubleshooting steps to check $SPLUNK_HOME/var/log/splunk/python.log for any error on each search head servers which are in SHC.