Splunk stopped sending Email for alerts and reports

[alexspunkshell]

Email server configuration was set up by Mail server team. Then i received mail for alerts and reports.

Now i am not receiving any mail for alerts and reports. When i check splunk logs i see 

ERROR:root:Connection unexpectedly closed while sending mail to alxxx&xxxx.com.

Please help here. How to solve this issue.

[isoutamo]

Hi

can you test it by this post https://ec.haxx.se/usingcurl/usingcurl-smtp just change those names etc to your splunk, your etc. And use curl -v 

r. Ismo

[alexspunkshell]

@isoutamo While running the curl command it show connected to smtp.xxxx.xxx port 80 and below.

Still splunk stopped sending emails.

[isoutamo · Accepted solution]

That said Status 403 access is denied. This means than you mail MTA is probably require authentication to allow connect and sending mail. Maybe someone has changed policy that unauthenticated mail sending is not allowed anymore?

[alexspunkshell]

@isoutamo Testing with gmail settings. Able to receive emails properly.

I hope issue with existing mail server authendication.

[thambisetty]

I believe there is issue with permissions in file system. can you check below:

what is the user under splunkd running?

if splunkd is running with non-root user, can you check all files under $SPLUNK_HOME are owned by non-root user or some of the files are owned by root where non-root user doesn't have permission to read for example alert_actions.conf 

can you run below 

splunk cmd btool alert_actions list email --debug | grep "mailserver"

you will see a path , 

ls -ltr <above path>

see the owner is matching with owner splunkd is running.

[alexspunkshell]

@thambisetty I am unable to search the below command. It shows"command not found"

[thambisetty]

$SPLUNK_HOME is where your splunk is installed for example /opt/splunk

$SPLUNK_HOME/bin/splunk cmd btool alert_actions list email --debug | grep "mailserver"

[alexspunkshell]

@thambisetty No luck. Its showing "No such file or directory"

[thambisetty]

can you telnet your smtp from Splunk box like below to see the connectivity?

login to SSH/RDP of your splunk 

open cmd 

telnet yoursmtphost 25

you should see message connected, other you don't have connectivity.

[alexspunkshell]

@thambisetty Telnet is fine. I see connected to smtp.xxxx.xxxx.

[thambisetty]

can you check email settings under Settings ->  Server Settings (Under System ) -> Email Settings.

you should see Mail host value and username and password. if values are not present, fill all the fields.

[alexspunkshell]

@thambisetty In Email Settings i see mail host details without port number. Also Email security is none. User name & pass is blank. We didn't changed any setting. Don't know why splunk stopped sendig mails