We were able to send mails till yesterday but from today we are facing the below error and mails are not getting triggered
ERROR sendemail:452 - Connection unexpectedly closed while sending mail to:
ERROR sendemail:137 - Sending email. subject="#CGO# Splunk Dashboard: 'CFS COS2 Cloud Reports'", results_link="http://ip-xx-xx-xx-xxxx:xx/app/search/@go?sid=", recipients="
Solution is provided here-
https://answers.splunk.com/answers/548703/why-are-mailserver-settings-in-alert-actionsconf-n.html?
from my observation if we save it through UI the savedsearch.conf of the app/local gets updated with action.email.mailserver = localhost (bug when edited from ui) instead in order to use the provided settings in email settings which is stored in alert_actions.conf in /etc/system/local you can either overwrite the ction.email.mailserver = in savedsearch.conf from command line or comment it and use the updated one from alert_actions.conf
This may be a SMTP exception problem. The Team managing the SMTP Virtual name/ Servers needs to add your host name to the exceptions list allowing that server the ability to send outbound email.
As you said, that you were receiving emails till yesterday did you check with your team that splunk server is still whitelisted in SMTP server? Sometimes what happens is they give a time limited access. So first confirm from SMTP guys.
your other options are getting a SMTP relay server setup where you can use that one Box (usually one of the server u manage) to send out the emails on your behalf (Opposed to the virtual name) downfall with this is it's a single point of failure where a Load Balanced SMTP virtual name... usually has more than one subnet represented its disaster recovery ready.
The last option is to setup Splunk as an SMTP server or another server locally that you manage.
If you work in an Enterprise or for a Gov. It's better and easier if you work this through your exchange team so that you have high availability with your email.
We are using gmail SMTP server here. smtp.gmail.com:465 is the SMTP details which we got from google.
Try sending email through SPL
<code>my search terms | sendemail to=foo@bar.com sendresults=true server=smtp.gmail.com:465
</code>
and see what error you get.