I have an alert_actions.conf file that is pushed out to our search heads via deployment server. All of the settings (hostname, mailserver, from) are being ignored when in the app context. If I move the same file into $SPLUNK_HOME/etc/system/local, everything works.
I ran "splunk cmd btool alert_actions list" and the output is identical no matter where I put alert_actions.conf. In both cases, it looks like the settings are correct.