Search Head Clustering: Why is the Deployer not deploying my email settings in alert_actions.conf properly to search heads?

[dturner83]

I've got an app called configuration. This app pushes authentication, outputs, and web conf files successfully to the 3 search heads. However alert_actions.conf, when deployed with the deployer in the same configuration app, it does not appear to deploy my email settings for alerting. The search heads continue to use the default settings (which are unconfigured) and email fails to send.

The alert_actions.conf file works properly on our stand alone search head which we are replacing so I know it's functional.

Does anyone know how to properly deploy this using the deployer?

[894859]

I was able to deploy alert_actions.conf to a SH Cluster but was only able to get the settings in that file to work if i also deployed a default.meta file within the same app with the following entry:
[alert_actions]
export = system

[meglin_splunk]

Also worked for me on 6.5.0

[hortonew]

Worked for me.

[gn694]

That worked for me too! Thank you!! I wish I had seen this 6 hours ago!!!
Any idea why this is required only for settings in alert_actions.conf? All of my other settings in the same app work fine.

[dturner83 · Accepted solution]

I cannot say this is the overall answer, but essentially how we've addressed this problem.

1. We removed alert_actions.conf from shcluster/apps/configuration app.
2. Since alert_actions.conf is in the whitelist for configuration settings, we made the proper changes on 1 search head for Email Settings through the GUI and those were properly deployed to the other search heads and it appears it's working well.

I did review the alert_actions.conf file produced by using the GUI and it's similar to the one we were trying to push, only differences were in where footer.text was placed.

I'd love to know why it didn't work with deployer, but I'm moving on from this one.

[jkat54]

This fixed it thanks! I had hard time reading what you said so let me clarify it for others.

To setup email on Search Head Clusters, log onto one of the search heads (any), and go to settings -> show all settings, settings -> server config -> email settings... define your settings and save.

It will propagate from there to all the other search heads.

Also make sure you remove the custom alert_actions.conf you may have already deployed while pulling you hair out as to why it doesnt work...

[kphillipson]

Thank you dturner83 and Jkat54. Its puzzling why they hide the email settings.

[dturner83]

True custom alert_actions.conf will make you pull your hair out.

Unfortunately based on the documentation though it seems as though you can deploy alert_actions.conf with the deployer inside an app, push with deployer, and it should update the search heads, but we've been unable to get this function to work. Changing in the GUI 'works', but if you're like us in our environment, we use a mixture of Chef for initial Splunk builds and the deployer for search head configuration updates, so using the GUI is less than ideal.

[esix_splunk]

After you do a deploy from the Deployer, is the file physically deployed to the SHC members? You also need to check your local directories on the SHC members, if you have created local configurations via the GUI, those will overwrite the deployed options. So you need to either delete the $splunk_home$/etc/apps/appname/local/alert_actions.conf, or merge those into your default on the deployer.

[dturner83]

Yes the file gets deployed to /etc/apps/configuration/default/alert_actions.conf. Local directories on the SHC members reveal just alert_actions.conf files inside /etc/apps/configuration/default and the default.old directories that shc seems to create.

Is this problem in how we are handling files on the deployer perhaps? apps no longer seem to have /local folders and just have /default now which seems a little strange.

[maciep]

with respect to that last apps question, the local folder on the the search heads are for changes made in the gui.

So when you apply the bundle from your deployer, splunk runs through the whole precedence algorithm for the apps on the deployer - local vs default. The resulting files then will end up getting saved in just default folder under the app on the search heads themselves.

If you then make changes in the splunk web for an app - create a search, update dashboard, create an extraction, whatever...those get saved to the local app folder on the search head and then they get replicated to the rest of the members.

So now changes made in splunk web don't get overwritten when you redeploy from the deployer. But you can still have local/default folders in your app on the deployer.

Hope that helps with that question at least. Not sure about alert config though...

[dturner83]

I deploy alert_actions.conf to /etc/shcluster/apps/configuration/ folder on the Deployer.

Then I apply shcluster-bundle to the shc members. This puts alert_actions.conf in /etc/apps/configuration/default/alert_actions.conf on the shc members, in the same manner as we use it for authentication, outputs, and web conf files, however doesn't seem to take effect. When looking at python.log I continue to get failures saying connection refused connecting to localhost to send these messages which would leave me to believe some alert_actions.conf exists from default that has higher precedence than an app potentially.

using find here are the files which match alert_actions.conf. I would expect /opt/splunk/etc/apps/configuration/default/alert_actions.conf to take precedence here.

/opt/splunk/etc/apps/configuration/default.old.20150813-163109/alert_actions.conf
/opt/splunk/etc/apps/configuration/default.old.20150813-155352/alert_actions.conf
/opt/splunk/etc/apps/configuration/default.old.20150813-163835/alert_actions.conf
/opt/splunk/etc/apps/configuration/default/alert_actions.conf
/opt/splunk/etc/system/default/alert_actions.conf

I then take a look using btool to determine what's getting used for alert_actions and the following appears:

splunk cmd btool alert_actions list

[default]
hostname =
maxresults = 10000
maxtime = 5m
track_alert = 0
ttl = 10p
[email]
auth_password =
auth_username =
bcc =
cc =
command = $action.email.preprocess_results{default=""}$ | sendemail "results_link=$results.url$" "ssname=$name$" "graceful=$graceful{default=True}$" "trigger_time=$trigger_time$" maxinputs="$action.email.maxresults{default=10000}$" maxtime="$action.email.maxtime{default=5m}$" results_file="$results.file$"
footer.text = If you believe you've received this email in error, please see your Splunk administrator.

splunk > the engine for machine data
format = table
from = SplunkEmail@mydomain.com
hostname = splunksearch.mydomain.com
include.results_link = 1
include.search = 0
include.trigger = 0
include.trigger_time = 0
include.view_link = 1
inline = 0
mailserver = mailrelay.mydomain.com
maxresults = 10000
maxtime = 5m
message.alert = The alert condition for '$name$' was triggered.
message.report = The scheduled report '$name$' has run.
pdfview =
preprocess_results =
priority = 3
reportCIDFontList = gb cns jp kor
reportIncludeSplunkLogo = 1
reportPaperOrientation = portrait
reportPaperSize = letter
reportServerEnabled = false
reportServerURL =
sendcsv = 0
sendpdf = 0
sendresults = 0
subject = Splunk Alert: $name$
subject.alert = Splunk Alert: $name$
subject.report = Splunk Report: $name$
to =
track_alert = 1
ttl = 86400
useNSSubject = 0
use_ssl = 0
use_tls = 0
width_sort_columns = 1
[populate_lookup]
command = copyresults dest="$action.populate_lookup.dest$" sid="$search_id$"
dest =
hostname =
maxresults = 10000
maxtime = 5m
track_alert = 0
ttl = 120
[rss]
command = createrss "path=$name$.xml" "name=$name$" "link=$results.url$" "descr=Alert trigger: $name$, results.count=$results.count$ " "count=30" "graceful=$graceful{default=1}$" maxtime="$action.rss.maxtime{default=1m}$"
hostname =
maxresults = 10000
maxtime = 1m
track_alert = 0
ttl = 86400
[script]
command = runshellscript "$action.script.filename$" "$results.count$" "$search$" "$search$" "$name$" "Saved Search [$name$] $counttype$($results.count$)" "$results.url$" "$deprecated_arg$" "$search_id$" "$results.file$" maxtime="$action.script.maxtime{default=5m}$"
filename =
hostname =
maxresults = 10000
maxtime = 5m
track_alert = 1
ttl = 600
[summary_index]
name = summary
command = summaryindex spool=t uselb=t addtime=t index="$action.summary_index._name{required=yes}$" file="$name_hash$$#random$.stash_new" name="$name$" marker="$action.summary_index*{format=$KEY=\\"$VAL\\", key_regex="action.summary_index.(?!(?:command|inline|maxresults|maxtime|ttl|track_alert|(?:_.))$)(.)"}$"
hostname =
inline = 1
maxresults = 10000
maxtime = 5m
track_alert = 0
ttl = 120

[esix_splunk]

Is this the incorrect alerts? Add '--debug' to the end of btool and it will tell you which file is being used for the parameters.

[dturner83]

It picks up some pieces of he appropriate alert_actions.conf file and it looks like it's picking up my file for some parts but logs indicate it's still trying to use localhost in the mail server hostname.

Here are the results from the --debug - Also thanks! I didn't know this was an option but is very helpful.

/opt/splunk/etc/system/default/alert_actions.conf [default]
/opt/splunk/etc/system/default/alert_actions.conf hostname =
/opt/splunk/etc/system/default/alert_actions.conf maxresults = 10000
/opt/splunk/etc/system/default/alert_actions.conf maxtime = 5m
/opt/splunk/etc/system/default/alert_actions.conf track_alert = 0
/opt/splunk/etc/system/default/alert_actions.conf ttl = 10p
/opt/splunk/etc/apps/configuration/default/alert_actions.conf [email]
/opt/splunk/etc/system/default/alert_actions.conf auth_password =
/opt/splunk/etc/system/default/alert_actions.conf auth_username =
/opt/splunk/etc/system/default/alert_actions.conf bcc =
/opt/splunk/etc/system/default/alert_actions.conf cc =
/opt/splunk/etc/system/default/alert_actions.conf command = $action.email.preprocess_results{default=""}$ | sendemail "results_link=$results.url$" "ssname=$name$" "graceful=$graceful{default=True}$" "trigger_time=$trigger_time$" maxinputs="$action.email.maxresults{default=10000}$" maxtime="$action.email.maxtime{default=5m}$" results_file="$results.file$"
/opt/splunk/etc/system/default/alert_actions.conf footer.text = If you believe you've received this email in error, please see your Splunk administrator.

splunk > the engine for machine data
/opt/splunk/etc/system/default/alert_actions.conf format = table
/opt/splunk/etc/apps/configuration/default/alert_actions.conf from = SplunkEmail@mydomain.com
/opt/splunk/etc/apps/configuration/default/alert_actions.conf hostname = splunksearch.mydomain.com
/opt/splunk/etc/system/default/alert_actions.conf include.results_link = 1
/opt/splunk/etc/system/default/alert_actions.conf include.search = 0
/opt/splunk/etc/system/default/alert_actions.conf include.trigger = 0
/opt/splunk/etc/system/default/alert_actions.conf include.trigger_time = 0
/opt/splunk/etc/system/default/alert_actions.conf include.view_link = 1
/opt/splunk/etc/system/default/alert_actions.conf inline = 0
/opt/splunk/etc/apps/configuration/default/alert_actions.conf mailserver = mailrelay.mydomain.com
/opt/splunk/etc/system/default/alert_actions.conf maxresults = 10000
/opt/splunk/etc/system/default/alert_actions.conf maxtime = 5m
/opt/splunk/etc/system/default/alert_actions.conf message.alert = The alert condition for '$name$' was triggered.
/opt/splunk/etc/system/default/alert_actions.conf message.report = The scheduled report '$name$' has run.
/opt/splunk/etc/system/default/alert_actions.conf pdfview =
/opt/splunk/etc/system/default/alert_actions.conf preprocess_results =
/opt/splunk/etc/system/default/alert_actions.conf priority = 3
/opt/splunk/etc/system/default/alert_actions.conf reportCIDFontList = gb cns jp kor
/opt/splunk/etc/system/default/alert_actions.conf reportIncludeSplunkLogo = 1
/opt/splunk/etc/system/default/alert_actions.conf reportPaperOrientation = portrait
/opt/splunk/etc/system/default/alert_actions.conf reportPaperSize = letter
/opt/splunk/etc/system/default/alert_actions.conf reportServerEnabled = false
/opt/splunk/etc/apps/configuration/default/alert_actions.conf reportServerURL =
/opt/splunk/etc/system/default/alert_actions.conf sendcsv = 0
/opt/splunk/etc/system/default/alert_actions.conf sendpdf = 0
/opt/splunk/etc/system/default/alert_actions.conf sendresults = 0
/opt/splunk/etc/system/default/alert_actions.conf subject = Splunk Alert: $name$
/opt/splunk/etc/system/default/alert_actions.conf subject.alert = Splunk Alert: $name$
/opt/splunk/etc/system/default/alert_actions.conf subject.report = Splunk Report: $name$
/opt/splunk/etc/system/default/alert_actions.conf to =
/opt/splunk/etc/system/default/alert_actions.conf track_alert = 1
/opt/splunk/etc/system/default/alert_actions.conf ttl = 86400
/opt/splunk/etc/system/default/alert_actions.conf useNSSubject = 0
/opt/splunk/etc/system/default/alert_actions.conf use_ssl = 0
/opt/splunk/etc/system/default/alert_actions.conf use_tls = 0
/opt/splunk/etc/system/default/alert_actions.conf width_sort_columns = 1
/opt/splunk/etc/system/default/alert_actions.conf [populate_lookup]
/opt/splunk/etc/system/default/alert_actions.conf command = copyresults dest="$action.populate_lookup.dest$" sid="$search_id$"
/opt/splunk/etc/system/default/alert_actions.conf dest =
/opt/splunk/etc/system/default/alert_actions.conf hostname =
/opt/splunk/etc/system/default/alert_actions.conf maxresults = 10000
/opt/splunk/etc/system/default/alert_actions.conf maxtime = 5m
/opt/splunk/etc/system/default/alert_actions.conf track_alert = 0
/opt/splunk/etc/system/default/alert_actions.conf ttl = 120
/opt/splunk/etc/system/default/alert_actions.conf [rss]
/opt/splunk/etc/system/default/alert_actions.conf command = createrss "path=$name$.xml" "name=$name$" "link=$results.url$" "descr=Alert trigger: $name$, results.count=$results.count$ " "count=30" "graceful=$graceful{default=1}$" maxtime="$action.rss.maxtime{default=1m}$"
/opt/splunk/etc/system/default/alert_actions.conf hostname =
/opt/splunk/etc/system/default/alert_actions.conf maxresults = 10000
/opt/splunk/etc/system/default/alert_actions.conf maxtime = 1m
/opt/splunk/etc/system/default/alert_actions.conf track_alert = 0
/opt/splunk/etc/system/default/alert_actions.conf ttl = 86400
/opt/splunk/etc/system/default/alert_actions.conf [script]
/opt/splunk/etc/system/default/alert_actions.conf command = runshellscript "$action.script.filename$" "$results.count$"
"$search$" "$search$" "$name$" "Saved Search [$name$] $counttype$($results.count$)" "$results.url$" "$deprecated_arg$" "$search_id$" "$results.file$" maxtime="$action.script.maxtime{default=5m}$"
/opt/splunk/etc/system/default/alert_actions.conf filename =
/opt/splunk/etc/system/default/alert_actions.conf hostname =
/opt/splunk/etc/system/default/alert_actions.conf maxresults = 10000
/opt/splunk/etc/system/default/alert_actions.conf maxtime = 5m
/opt/splunk/etc/system/default/alert_actions.conf track_alert = 1
/opt/splunk/etc/system/default/alert_actions.conf ttl = 600
/opt/splunk/etc/system/default/alert_actions.conf [summary_index]
/opt/splunk/etc/system/default/alert_actions.conf name = summary
/opt/splunk/etc/system/default/alert_actions.conf command = summaryindex spool=t uselb=t addtime=t index="$action.summary_index._name{required=yes}$" file="$name_hash$$#random$.stash_new" name="$name$" marker="$action.summary_index*{format=$KEY=\\"$VAL\\", key_regex="action.summary_index.(?!(?:command|inline|maxresults|maxtime|ttl|track_alert|(?:_.))$)(.)"}$"
/opt/splunk/etc/system/default/alert_actions.conf hostname =
/opt/splunk/etc/system/default/alert_actions.conf inline = 1
/opt/splunk/etc/system/default/alert_actions.conf maxresults = 10000
/opt/splunk/etc/system/default/alert_actions.conf maxtime = 5m
/opt/splunk/etc/system/default/alert_actions.conf track_alert = 0
/opt/splunk/etc/system/default/alert_actions.conf ttl = 120