Why are the email alerts not being sent anymore?

[nick405060]

Alerts no longer email, however they do show up in triggered alerts. This started sometime yesterday, before that we have been getting email alerts and creating email alerts for months.

var\log\splunk\python.log shows:

2018-08-16 15:44:41,450 PDT ERROR sendemail:115 - Sending email. subject="testalert010", results_link="http://ABCSPLNK02:8000/app/abc_ist/search?q=%7Cloadjob%20rt_scheduler__username_ZHd0X2lzdA__testtest_at_1534459452_73.0%20%7C%20head%201%20%7C%20tail%201&earliest=0&latest=now", recipients="[u'myemailaddress@abc.com']", server="localhost" 
2018-08-16 15:44:41,450 PDT ERROR sendemail:378 - [Errno 10061] No connection could be made because the target machine actively refused it while sending mail to: u'myemailaddress@abc.com 

We've checked and there is no firewall/exchange/mimecast/smtp blocking the emails. I have been messing with limits.conf, savedsearches.conf, and alert_actions.conf for completely separate reasons, and those changes should not have affected email alerting. I went back and deleted every one of those mentioned .conf files that have been edited in the last three days (in the local directories, so the default files have taken back over) and still no luck.

I also installed the Slack app in the last three days, is it possible that the installation overrode SMTP settings in Splunk or something? (I have since deleted the Slack app in case that was the reason why, without it fixing the problem)

Have looked at all Splunk answers for this issue to no avail.

[nick405060 · Accepted solution]

Posted this question so I could answer it for the Splunk community. As explained by a Splunk support engineer:

My recommendation is to set the server back to "smtp.abc.com" by navigating in the GUI Settings->Server settings->Email settings 
Then change the "Mail host" field from localhost, to smtp.abc.com. 

Not sure how it got changed, and it's interesting that the change was not reflected in any config files anywhere, so hours of searching in PowerShell wasn't able to detect the change. My theory is that installing our Slack integration app defaulted the mail host back to localhost

[abhijitsaoji]

Hi, I am facing same issue, my real time alert is not working at all. It is neither appearing in the triggered alerts nor sending any emails. I have changed the alert type to scheduled - every hour on 30th minute and worked like a charm. Not sure what would be the issue with real time, I have read few comments about latency and ran the query supplied but latency is coming in seconds so probably it can be ruled out. any other thoughts, please let me know.