Alerting
Highlighted

Search Head Clustering: Why is the Deployer not deploying my email settings in alert_actions.conf properly to search heads?

Path Finder

I've got an app called configuration. This app pushes authentication, outputs, and web conf files successfully to the 3 search heads. However alert_actions.conf, when deployed with the deployer in the same configuration app, it does not appear to deploy my email settings for alerting. The search heads continue to use the default settings (which are unconfigured) and email fails to send.

The alert_actions.conf file works properly on our stand alone search head which we are replacing so I know it's functional.

Does anyone know how to properly deploy this using the deployer?

Highlighted

Re: Search Head Clustering: Why is the Deployer not deploying my email settings in alert_actions.conf properly to search heads?

Splunk Employee
Splunk Employee

After you do a deploy from the Deployer, is the file physically deployed to the SHC members? You also need to check your local directories on the SHC members, if you have created local configurations via the GUI, those will overwrite the deployed options. So you need to either delete the $splunkhome$/etc/apps/appname/local/alertactions.conf, or merge those into your default on the deployer.

0 Karma
Highlighted

Re: Search Head Clustering: Why is the Deployer not deploying my email settings in alert_actions.conf properly to search heads?

Path Finder

I deploy alert_actions.conf to /etc/shcluster/apps/configuration/ folder on the Deployer.

Then I apply shcluster-bundle to the shc members. This puts alertactions.conf in /etc/apps/configuration/default/alertactions.conf on the shc members, in the same manner as we use it for authentication, outputs, and web conf files, however doesn't seem to take effect. When looking at python.log I continue to get failures saying connection refused connecting to localhost to send these messages which would leave me to believe some alert_actions.conf exists from default that has higher precedence than an app potentially.

using find here are the files which match alertactions.conf. I would expect /opt/splunk/etc/apps/configuration/default/alertactions.conf to take precedence here.

/opt/splunk/etc/apps/configuration/default.old.20150813-163109/alertactions.conf
/opt/splunk/etc/apps/configuration/default.old.20150813-155352/alert
actions.conf
/opt/splunk/etc/apps/configuration/default.old.20150813-163835/alertactions.conf
/opt/splunk/etc/apps/configuration/default/alert
actions.conf
/opt/splunk/etc/system/default/alert_actions.conf

I then take a look using btool to determine what's getting used for alert_actions and the following appears:

splunk cmd btool alert_actions list

[default]
hostname =
maxresults = 10000
maxtime = 5m
trackalert = 0
ttl = 10p
[email]
auth
password =
authusername =
bcc =
cc =
command = $action.email.preprocess
results{default=""}$ | sendemail "resultslink=$results.url$" "ssname=$name$" "graceful=$graceful{default=True}$" "triggertime=$triggertime$" maxinputs="$action.email.maxresults{default=10000}$" maxtime="$action.email.maxtime{default=5m}$" resultsfile="$results.file$"
footer.text = If you believe you've received this email in error, please see your Splunk administrator.

splunk > the engine for machine data
format = table
from = SplunkEmail@mydomain.com
hostname = splunksearch.mydomain.com
include.resultslink = 1
include.search = 0
include.trigger = 0
include.trigger
time = 0
include.viewlink = 1
inline = 0
mailserver = mailrelay.mydomain.com
maxresults = 10000
maxtime = 5m
message.alert = The alert condition for '$name$' was triggered.
message.report = The scheduled report '$name$' has run.
pdfview =
preprocess
results =
priority = 3
reportCIDFontList = gb cns jp kor
reportIncludeSplunkLogo = 1
reportPaperOrientation = portrait
reportPaperSize = letter
reportServerEnabled = false
reportServerURL =
sendcsv = 0
sendpdf = 0
sendresults = 0
subject = Splunk Alert: $name$
subject.alert = Splunk Alert: $name$
subject.report = Splunk Report: $name$
to =
trackalert = 1
ttl = 86400
useNSSubject = 0
use
ssl = 0
usetls = 0
width
sortcolumns = 1
[populate
lookup]
command = copyresults dest="$action.populatelookup.dest$" sid="$searchid$"
dest =
hostname =
maxresults = 10000
maxtime = 5m
trackalert = 0
ttl = 120
[rss]
command = createrss "path=$name$.xml" "name=$name$" "link=$results.url$" "descr=Alert trigger: $name$, results.count=$results.count$ " "count=30" "graceful=$graceful{default=1}$" maxtime="$action.rss.maxtime{default=1m}$"
hostname =
maxresults = 10000
maxtime = 1m
track
alert = 0
ttl = 86400
[script]
command = runshellscript "$action.script.filename$" "$results.count$" "$search$" "$search$" "$name$" "Saved Search [$name$] $counttype$($results.count$)" "$results.url$" "$deprecatedarg$" "$searchid$" "$results.file$" maxtime="$action.script.maxtime{default=5m}$"
filename =
hostname =
maxresults = 10000
maxtime = 5m
trackalert = 1
ttl = 600
[summary
index]
name = summary
command = summaryindex spool=t uselb=t addtime=t index="$action.summary
index.name{required=yes}$" file="$namehash$$#random$.stashnew" name="$name$" marker="$action.summaryindex*{format=$KEY=\\"$VAL\\", keyregex="action.summaryindex.(?!(?:command|inline|maxresults|maxtime|ttl|trackalert|(?:.))$)(.)"}$"
hostname =
inline = 1
maxresults = 10000
maxtime = 5m
track
alert = 0
ttl = 120

Highlighted

Re: Search Head Clustering: Why is the Deployer not deploying my email settings in alert_actions.conf properly to search heads?

Splunk Employee
Splunk Employee

Is this the incorrect alerts? Add '--debug' to the end of btool and it will tell you which file is being used for the parameters.

0 Karma
Highlighted

Re: Search Head Clustering: Why is the Deployer not deploying my email settings in alert_actions.conf properly to search heads?

Path Finder

It picks up some pieces of he appropriate alert_actions.conf file and it looks like it's picking up my file for some parts but logs indicate it's still trying to use localhost in the mail server hostname.

Here are the results from the --debug - Also thanks! I didn't know this was an option but is very helpful.

/opt/splunk/etc/system/default/alertactions.conf [default]
/opt/splunk/etc/system/default/alert
actions.conf hostname =
/opt/splunk/etc/system/default/alertactions.conf maxresults = 10000
/opt/splunk/etc/system/default/alert
actions.conf maxtime = 5m
/opt/splunk/etc/system/default/alertactions.conf trackalert = 0
/opt/splunk/etc/system/default/alertactions.conf ttl = 10p
/opt/splunk/etc/apps/configuration/default/alert
actions.conf [email]
/opt/splunk/etc/system/default/alertactions.conf authpassword =
/opt/splunk/etc/system/default/alertactions.conf authusername =
/opt/splunk/etc/system/default/alertactions.conf bcc =
/opt/splunk/etc/system/default/alert
actions.conf cc =
/opt/splunk/etc/system/default/alertactions.conf command = $action.email.preprocessresults{default=""}$ | sendemail "resultslink=$results.url$" "ssname=$name$" "graceful=$graceful{default=True}$" "triggertime=$triggertime$" maxinputs="$action.email.maxresults{default=10000}$" maxtime="$action.email.maxtime{default=5m}$" resultsfile="$results.file$"
/opt/splunk/etc/system/default/alert_actions.conf footer.text = If you believe you've received this email in error, please see your Splunk administrator.

splunk > the engine for machine data
/opt/splunk/etc/system/default/alertactions.conf format = table
/opt/splunk/etc/apps/configuration/default/alert
actions.conf from = SplunkEmail@mydomain.com
/opt/splunk/etc/apps/configuration/default/alertactions.conf hostname = splunksearch.mydomain.com
/opt/splunk/etc/system/default/alert
actions.conf include.resultslink = 1
/opt/splunk/etc/system/default/alert
actions.conf include.search = 0
/opt/splunk/etc/system/default/alertactions.conf include.trigger = 0
/opt/splunk/etc/system/default/alert
actions.conf include.triggertime = 0
/opt/splunk/etc/system/default/alert
actions.conf include.viewlink = 1
/opt/splunk/etc/system/default/alert
actions.conf inline = 0
/opt/splunk/etc/apps/configuration/default/alertactions.conf mailserver = mailrelay.mydomain.com
/opt/splunk/etc/system/default/alert
actions.conf maxresults = 10000
/opt/splunk/etc/system/default/alertactions.conf maxtime = 5m
/opt/splunk/etc/system/default/alert
actions.conf message.alert = The alert condition for '$name$' was triggered.
/opt/splunk/etc/system/default/alertactions.conf message.report = The scheduled report '$name$' has run.
/opt/splunk/etc/system/default/alert
actions.conf pdfview =
/opt/splunk/etc/system/default/alertactions.conf preprocessresults =
/opt/splunk/etc/system/default/alertactions.conf priority = 3
/opt/splunk/etc/system/default/alert
actions.conf reportCIDFontList = gb cns jp kor
/opt/splunk/etc/system/default/alertactions.conf reportIncludeSplunkLogo = 1
/opt/splunk/etc/system/default/alert
actions.conf reportPaperOrientation = portrait
/opt/splunk/etc/system/default/alertactions.conf reportPaperSize = letter
/opt/splunk/etc/system/default/alert
actions.conf reportServerEnabled = false
/opt/splunk/etc/apps/configuration/default/alertactions.conf reportServerURL =
/opt/splunk/etc/system/default/alert
actions.conf sendcsv = 0
/opt/splunk/etc/system/default/alertactions.conf sendpdf = 0
/opt/splunk/etc/system/default/alert
actions.conf sendresults = 0
/opt/splunk/etc/system/default/alertactions.conf subject = Splunk Alert: $name$
/opt/splunk/etc/system/default/alert
actions.conf subject.alert = Splunk Alert: $name$
/opt/splunk/etc/system/default/alertactions.conf subject.report = Splunk Report: $name$
/opt/splunk/etc/system/default/alert
actions.conf to =
/opt/splunk/etc/system/default/alertactions.conf trackalert = 1
/opt/splunk/etc/system/default/alertactions.conf ttl = 86400
/opt/splunk/etc/system/default/alert
actions.conf useNSSubject = 0
/opt/splunk/etc/system/default/alertactions.conf usessl = 0
/opt/splunk/etc/system/default/alertactions.conf usetls = 0
/opt/splunk/etc/system/default/alertactions.conf widthsortcolumns = 1
/opt/splunk/etc/system/default/alert
actions.conf [populatelookup]
/opt/splunk/etc/system/default/alert
actions.conf command = copyresults dest="$action.populatelookup.dest$" sid="$searchid$"
/opt/splunk/etc/system/default/alertactions.conf dest =
/opt/splunk/etc/system/default/alert
actions.conf hostname =
/opt/splunk/etc/system/default/alertactions.conf maxresults = 10000
/opt/splunk/etc/system/default/alert
actions.conf maxtime = 5m
/opt/splunk/etc/system/default/alertactions.conf trackalert = 0
/opt/splunk/etc/system/default/alertactions.conf ttl = 120
/opt/splunk/etc/system/default/alert
actions.conf [rss]
/opt/splunk/etc/system/default/alertactions.conf command = createrss "path=$name$.xml" "name=$name$" "link=$results.url$" "descr=Alert trigger: $name$, results.count=$results.count$ " "count=30" "graceful=$graceful{default=1}$" maxtime="$action.rss.maxtime{default=1m}$"
/opt/splunk/etc/system/default/alert
actions.conf hostname =
/opt/splunk/etc/system/default/alertactions.conf maxresults = 10000
/opt/splunk/etc/system/default/alert
actions.conf maxtime = 1m
/opt/splunk/etc/system/default/alertactions.conf trackalert = 0
/opt/splunk/etc/system/default/alertactions.conf ttl = 86400
/opt/splunk/etc/system/default/alert
actions.conf [script]
/opt/splunk/etc/system/default/alertactions.conf command = runshellscript "$action.script.filename$" "$results.count$"
"$search$" "$search$" "$name$" "Saved Search [$name$] $counttype$($results.count$)" "$results.url$" "$deprecated
arg$" "$searchid$" "$results.file$" maxtime="$action.script.maxtime{default=5m}$"
/opt/splunk/etc/system/default/alert
actions.conf filename =
/opt/splunk/etc/system/default/alertactions.conf hostname =
/opt/splunk/etc/system/default/alert
actions.conf maxresults = 10000
/opt/splunk/etc/system/default/alertactions.conf maxtime = 5m
/opt/splunk/etc/system/default/alert
actions.conf trackalert = 1
/opt/splunk/etc/system/default/alert
actions.conf ttl = 600
/opt/splunk/etc/system/default/alertactions.conf [summaryindex]
/opt/splunk/etc/system/default/alertactions.conf _name = summary
/opt/splunk/etc/system/default/alert
actions.conf command = summaryindex spool=t uselb=t addtime=t index="$action.summaryindex.name{required=yes}$" file="$namehash$$#random$.stashnew" name="$name$" marker="$action.summaryindex{format=$KEY=\\"$VAL\\", keyregex="action.summaryindex.(?!(?:command|inline|maxresults|maxtime|ttl|trackalert|(?:.))$)(.*)"}$"
/opt/splunk/etc/system/default/alertactions.conf hostname =
/opt/splunk/etc/system/default/alert
actions.conf inline = 1
/opt/splunk/etc/system/default/alertactions.conf maxresults = 10000
/opt/splunk/etc/system/default/alert
actions.conf maxtime = 5m
/opt/splunk/etc/system/default/alertactions.conf trackalert = 0
/opt/splunk/etc/system/default/alert_actions.conf ttl = 120

0 Karma
Highlighted

Re: Search Head Clustering: Why is the Deployer not deploying my email settings in alert_actions.conf properly to search heads?

Path Finder

Yes the file gets deployed to /etc/apps/configuration/default/alertactions.conf. Local directories on the SHC members reveal just alertactions.conf files inside /etc/apps/configuration/default and the default.old directories that shc seems to create.

Is this problem in how we are handling files on the deployer perhaps? apps no longer seem to have /local folders and just have /default now which seems a little strange.

0 Karma
Highlighted

Re: Search Head Clustering: Why is the Deployer not deploying my email settings in alert_actions.conf properly to search heads?

Champion

with respect to that last apps question, the local folder on the the search heads are for changes made in the gui.

So when you apply the bundle from your deployer, splunk runs through the whole precedence algorithm for the apps on the deployer - local vs default. The resulting files then will end up getting saved in just default folder under the app on the search heads themselves.

If you then make changes in the splunk web for an app - create a search, update dashboard, create an extraction, whatever...those get saved to the local app folder on the search head and then they get replicated to the rest of the members.

So now changes made in splunk web don't get overwritten when you redeploy from the deployer. But you can still have local/default folders in your app on the deployer.

Hope that helps with that question at least. Not sure about alert config though...

0 Karma
Highlighted

Re: Search Head Clustering: Why is the Deployer not deploying my email settings in alert_actions.conf properly to search heads?

Path Finder

I cannot say this is the overall answer, but essentially how we've addressed this problem.

  1. We removed alert_actions.conf from shcluster/apps/configuration app.
  2. Since alert_actions.conf is in the whitelist for configuration settings, we made the proper changes on 1 search head for Email Settings through the GUI and those were properly deployed to the other search heads and it appears it's working well.

I did review the alert_actions.conf file produced by using the GUI and it's similar to the one we were trying to push, only differences were in where footer.text was placed.

I'd love to know why it didn't work with deployer, but I'm moving on from this one.

View solution in original post

Highlighted

Re: Search Head Clustering: Why is the Deployer not deploying my email settings in alert_actions.conf properly to search heads?

SplunkTrust
SplunkTrust

This fixed it thanks! I had hard time reading what you said so let me clarify it for others.

To setup email on Search Head Clusters, log onto one of the search heads (any), and go to settings -> show all settings, settings -> server config -> email settings... define your settings and save.

It will propagate from there to all the other search heads.

Also make sure you remove the custom alert_actions.conf you may have already deployed while pulling you hair out as to why it doesnt work...

0 Karma
Highlighted

Re: Search Head Clustering: Why is the Deployer not deploying my email settings in alert_actions.conf properly to search heads?

Path Finder

True custom alert_actions.conf will make you pull your hair out.

Unfortunately based on the documentation though it seems as though you can deploy alert_actions.conf with the deployer inside an app, push with deployer, and it should update the search heads, but we've been unable to get this function to work. Changing in the GUI 'works', but if you're like us in our environment, we use a mixture of Chef for initial Splunk builds and the deployer for search head configuration updates, so using the GUI is less than ideal.

0 Karma