Does anybody know where the failures of sendemail are being logged? I wonder about cases where the e-mail address no longer exists and what type of error is generated and where. _internal and _audit don't seem to have this data.
There are two possible cases here.
1) The sendemail command (or the equivalent alert action) is unable to submit the email for delivery to the immediate SMTP server (due to bad/lack of authentication, network problems and so on). Those kinds of problems will be reported as logs from sendemail.py as @nyc_jason already showed
2) The email is properly submitted to the SMTP server but the delivery process doesn't complete properly (due to one of the many possible problems that can happen in email path) - well, then you have to troubleshoot your email system just like you would do with any other email. If the email generated from Splunk has some deliverable From address configured you might want to check the corresponding mailbox to see whether there were no delivery problem reports generated.
Are you looking for logs from your actual mail transfer agent (aka SMTP server) or an existing source in Splunk? Unless you actually ingest mail log, it won't be available.
When you say "e-mail address no longer exists," you don't mean that outlook.com used to exist but no longer, but a user's mailbox used to exist but no longer. Is this correct? Unless the server is rejecting connection (e.g., outlook.com all in a sudden stopped), Splunk submits data and will have no knowledge about mail handling. Only the MTA log will contain what you needed.
Great. What sort of errors _does_ sendemail report on?
try this:
index=_internal source=*python.log sendemail
Thank you, but unfortunately it doesn't show the failures.