Alerts triggered 30 times and only 3 emails received

[mufthmu]

Hi fellow splunkers,

I faced a mysterious issue where the number of triggered alerts do not match the number of emails received. When I check python.log, I see the alert is giving me this error

2020-09-25 18:49:01,765 +0000 ERROR     sendemail:142 - Sending email. subject="Splunk Alert: to be deleted", results_link="http://aws-prod-east-splunk.megh.thingspace.com/app/search/@go?sid=scheduler__admin__search__RMD57f4b1593a5b5364b_at_1601059740_8497_BA4F469F-14CB-4CBF-A20F-40A798E7F698", recipients="[u'myemail@email.com']", server="top-smtp-proxy.ts-prod.cloud:587"

2020-09-25 18:49:01,765 +0000 ERROR     sendemail:475 - (530, 'Authentication required', u'no-reply-top@verizon.com') while sending mail to: myemail@email.com

AND, I found this anomaly in my alert configuration. 

Note that sendemail command from search bar worked and I did receive the email. So it's only giving me error for alerts or scheduled searches.

Anyone else having this issue? 

[isoutamo]

This error means that receiving MTA cannot get correct user + password from splunk when it’s trying to send email to someone.
Why it’s working when you are sending it from GUI is interesting question,...

[mufthmu]

@isoutamo Thank you or the response, Although I'm not sure if it's about user + password issue simply because the exact same alert is still able to send email when triggered. But only small percentage of those triggered alerts are sent, the rest have that error I mentioned above.

I however, use app to put my alerts in and this is the alert_actions.conf file in system/local:

[email]
hostname = http://aws-prod-east-splunk.megh.thingspace.verizon.com
mailserver = top-smtp-proxy.ts-prod.cloud:587
pdf.header_left = none
pdf.header_right = none
disabled = 0
auth_password = {encrypted}
auth_username = AKIAUN3SJVAQRIOJW62G
from = myemail@mail.com (whitelisted)
use_tls = 1

and this is the alert_actions.conf in each app (I have about 10 app):

[email]
subject= |prod-us-east-1| SplunkAlert: $name$ $result.cid$