Where is "action.email.from = splunk@mycompany.com" in savedsearches.conf coming from?

[rgsage]

On Splunk 6.2.1.
We have many email alerts that are working fine (email alerts arrive as expected from monitor@mycompany.com). However one of our alerts sends email from splunk@mycompany.com. I do notice that whenever I make edits from the Splunk Web UI to this alert and click done, if I look in etc/apps/search/local/savedsearches.conf I see a line like:

action.email.from = splunk@mycompany.com

I don't want this. There is no 'from' field in Splunk Web. If manually remove this line from the savedsearches.conf file, it is restored after my next edit from Splunk Web.

Our etc/system/local/alert_actions.conf is like:

[email]
auth_password = **
auth_username = me@mycompany.com
mailserver = smtp.sendgrid.net:465
use_ssl = 1
hostname = splunk.corp.mycompany.com
from = monitor@mycompany.com

What am I missing? How can I set this alert to go out from monitor@mycompany.com like all our other alerts?

I don't see anything different in the alert config, but including redacted version here for completeness:

[My Alert]
action.email = 1
action.email.from = splunk@mycompany.com  # this line keeps coming back
action.email.inline = 1
action.email.message.alert = Dev Spike needs attention!
action.email.reportServerEnabled = 0
action.email.sendresults = 1
action.email.subject.alert = My Alert
action.email.to = me@mycompany.com
action.email.useNSSubject = 1
alert.suppress = 1
alert.suppress.period = 60m
alert.track = 1
counttype = number of events
cron_schedule = */1 * * * *
description = My Description
dispatch.earliest_time = @m-1m
dispatch.latest_time = @m
display.events.fields = ["host","source","sourcetype","severity"]
display.visualizations.chartHeight = 908
enableSched = 1
quantity = 0
relation = greater than
request.ui_dispatch_app = search
request.ui_dispatch_view = search
search = ***

[rgsage · Accepted solution]

Thanks to Splunk tech support for solving this one for us, and apologies to anyone in Splunk Answers land who spent time trying to figure it out from the data presented - because I edited the offending data in my post. The problem was my actual 'action.email.to' looked more like this:

action.email.to=me@mycompany.com,\n
someone_else@mycompany.com,\n
mylist@mycompany.com

The linebreaks caused the unexpected behavior when sending email and saving the alert from Web UI. The Web UI inline help does state that email addresses should be 'comma separated', however comma + space works fine, so I would consider this a Splunk bug. And difficult to spot because linebreaks and word wrap in the email to field look pretty similar. But I'm happy now, all our alerts working fine.

[rgsage]

Update on this with additional interesting tidbits...
If I manually update [My Alert] in etc/users/robert/search/local/savedsearches.conf, set action.email.from = monitor@mycompany.com, then:

• Use Web UI to Clone [My Alert]: the cloned alert has action.email.from = splunk@mycompany.com
• Use Web UI to edit [My Alert]: Splunk reverts [My Alert] to action.email.from = splunk@mycompany.com

We have many Splunk Reports that are schedule for nightly email delivery, and they ALL go out from monitor@mycompany.com

Minor correction to original post: Some Alert emails go out with no sender at all. The behavior is consistent per alert - that is the alerts that go out from splunk@mycompany.com or [no sender] always do so. Some Alerts go to:
monitor@mycompany.com (as specified in etc/system/local/alert_actions.conf)
splunk@mycompany.com (not sure why)
no sender

[woodcock]

As an admin go to Settings -> Server settings -> Email settings -> Send emails as and set to proper value.

[DalJeanis]

http://docs.splunk.com/Documentation/Splunk/6.5.3/Alert/Emailnotification#Configure_email_notificati...

Specify Email Format settings.
Link hostname. The hostname for outgoing results URLs. Enclose IPv6 addresses in square brackets. For example, use [2001:db8:0:1].
Send emails as. (Optional) Specify a sender identification, used in the "From" email header field. Use an email address or a string. Strings are concatenated with "@", using the hostname specified in alert_actions.conf for the machine sending the email notification or "@localhost" if no hostname is specified. Defaults to "splunk@" or "splunk@localhost" if no hostname is specified.
Email footer. Footer for all emails. Use text and/or tokens.

[DalJeanis]

Bottom line... of you don't want it to come back that way, tell it what it SHOULD be.

[rgsage]

Thank you for your quick reply. In case I wasn't clear we have done Settings > Server settings > Email settings as described in http://docs.splunk.com/Documentation/Splunk/6.5.3/Alert/Emailnotification#Configure_email_notificati...
The values we set there are apparently saved by Splunk in the etc/system/local/alert_actions.conf which I listed above, most importantly the 'from = monitor@mycompany.com'

All our other alerts go out from 'monitor@mycompany.com' except this one which goes out from 'splunk@mycompany.com'. There must be something special about this alert but I cannot see any difference. It may be older than the other alerts, created at a time when we did have 'from = splunk@mycompany.com' but I don't see how that would matter now...