We have 4 servers in a search head cluster. When we receive Splunk alerts from 3 out of 4 servers, they are displayed as received From "Splunk Alert". Emails from the last server are displayed as From splunk@hostname
All 4 servers have identical $SPLUNK_HOME/etc/system/default/alert_actions.conf and local/alert_actions.conf files:
1) ...default/alert_actions.conf:
"...# from email address (name only, host will be appended automatically from mailserver)
from=splunk
subject = Splunk Alert: $name$
subject.alert = Splunk Alert: $name$
subject.report = Splunk Report: $name$
useNSSubject = 0"
2) ...local/alert_actions.conf:
[email]
from = splunk
pdf.header_left = none
pdf.header_right = none
Any ideas what might cause this situation? Our goal to receive emails from all 4 servers as from "Splunk Alert"
I've contacted our messaging team, explained the issue and as they said "it's easy to fix". They added that email address to Contact "Splunk Alert".
The from in the email stanza defaults to splunk@$LOCALHOST but you can set it to anything. To have them send from the same address, just set them all to splunk@yourdomain. You can't set it through the UI in a cluster, it has to be done on the filesystem, but it works for us.
I've contacted our messaging team, explained the issue and as they said "it's easy to fix". They added that email address to Contact "Splunk Alert".
A stab in the dark: Does your email client's contact list know one of the email addresses as the full name "Splunk Alert"? If so, teach it the other emails as well.
@martin_mueller , and you are actually right about it.
After going through all the config files and comparing them on all 4 servers, checking os mail setting and mail logs without success, I came to the same conclusion as you! I've contacted our messaging team, explained the issue and as they said "it's easy to fix". They added that email address to Contact "Splunk Alert". Unfortunately, cannot force the alert to be sent from the server in question due few reasons, so waiting to a get a alert from it to confirm that it was solved
\o/
Compare the $SPLUNK_HOME/etc/system/local/alert_actions.conf files. That's where the difference is hiding.
Never change anything in a 'default' directory.
@richgalloway $SPLUNK_HOME/etc/system/local/alert_actions.conf files are also the same (just updated the question) on all 4 servers
Or run the btool command on alert_actions.conf with debug option to see what and where is the difference.
$SPLUNK_HOME/bin/splunk cmd btool alert_actions list --debug
@somesoni2, no difference found by running btool 😞