Alerting

Splunk stopped sending Email for alerts and reports

alexspunkshell
Contributor

Email server configuration was set up by Mail server team. Then i received mail for alerts and reports.

Now i am not receiving any mail for alerts and reports. When i check splunk logs i see 

ERROR:root:Connection unexpectedly closed while sending mail to alxxx&xxxx.com.

Please help here. How to solve this issue.

Labels (3)
Tags (3)
0 Karma
1 Solution

isoutamo
SplunkTrust
SplunkTrust
That said Status 403 access is denied. This means than you mail MTA is probably require authentication to allow connect and sending mail. Maybe someone has changed policy that unauthenticated mail sending is not allowed anymore?

View solution in original post

0 Karma

isoutamo
SplunkTrust
SplunkTrust

Hi

can you test it by this post https://ec.haxx.se/usingcurl/usingcurl-smtp just change those names etc to your splunk, your etc. And use curl -v 

r. Ismo

0 Karma

alexspunkshell
Contributor

@isoutamo While running the curl command it show connected to smtp.xxxx.xxx port 80 and below.

Still splunk stopped sending emails.

alexspunkshell_0-1599114453980.png

 

0 Karma

isoutamo
SplunkTrust
SplunkTrust
That said Status 403 access is denied. This means than you mail MTA is probably require authentication to allow connect and sending mail. Maybe someone has changed policy that unauthenticated mail sending is not allowed anymore?
0 Karma

alexspunkshell
Contributor

@isoutamo Testing with gmail settings. Able to receive emails properly.

I hope issue with existing mail server authendication.

0 Karma

thambisetty
SplunkTrust
SplunkTrust

I believe there is issue with permissions in file system. can you check below:

what is the user under splunkd running?

if splunkd is running with non-root user, can you check all files under $SPLUNK_HOME are owned by non-root user or some of the files are owned by root where non-root user doesn't have permission to read for example alert_actions.conf 

can you run below 

splunk cmd btool alert_actions list email --debug | grep "mailserver"

you will see a path , 

ls -ltr <above path>

see the owner is matching with owner splunkd is running.

————————————
If this helps, give a like below.
0 Karma

alexspunkshell
Contributor

@thambisetty I am unable to search the below command. It shows"command not found"

0 Karma

thambisetty
SplunkTrust
SplunkTrust

$SPLUNK_HOME is where your splunk is installed for example /opt/splunk

$SPLUNK_HOME/bin/splunk cmd btool alert_actions list email --debug | grep "mailserver"

————————————
If this helps, give a like below.
0 Karma

alexspunkshell
Contributor

@thambisetty No luck. Its showing "No such file or directory"

0 Karma

thambisetty
SplunkTrust
SplunkTrust

can you telnet your smtp from Splunk box like below to see the connectivity?

login to SSH/RDP of your splunk 

open cmd 

telnet yoursmtphost 25

you should see message connected, other you don't have connectivity.

————————————
If this helps, give a like below.
0 Karma

alexspunkshell
Contributor

@thambisetty Telnet is fine. I see connected to smtp.xxxx.xxxx.

0 Karma

thambisetty
SplunkTrust
SplunkTrust

can you check email settings under Settings ->  Server Settings (Under System ) -> Email Settings.

you should see Mail host value and username and password. if values are not present, fill all the fields.

————————————
If this helps, give a like below.
0 Karma

alexspunkshell
Contributor

@thambisetty In Email Settings i see mail host details without port number. Also Email security is none. User name & pass is blank. We didn't changed any setting. Don't know why splunk stopped sendig mails

0 Karma
Get Updates on the Splunk Community!

.conf24 | Day 0

Hello Splunk Community! My name is Chris, and I'm based in Canberra, Australia's capital, and I travelled for ...

Enhance Security Visibility with Splunk Enterprise Security 7.1 through Threat ...

 (view in My Videos)Struggling with alert fatigue, lack of context, and prioritization around security ...

Troubleshooting the OpenTelemetry Collector

  In this tech talk, you’ll learn how to troubleshoot the OpenTelemetry collector - from checking the ...