How to properly configure a deployment app to deploy configuration files

[avalle]

Hello,
I have searched the answer but can't seem to find out how to properly set up my apps in the deployment server.
I already have over 4k devices communicating to the deployment server, they are all in their correct server class and I have a directory for each app in the $SPLUNK_HOME/etc/deployment-apps. Let's say I have App1 and i want to use App1 to send my configuration files to a server class. I create $SPLUNK_HOME/etc/deployment-apps/App1/local/outputs.conf with the correct settings and I deploy the app but for some reason it does not update the outpts.conf on the clients. What am I doing wrong?

[woodcock]

If the problem is that you are deploying an outputs.conf from the DS and some (or all) of the forwarders do not begin to send differently (to the new indexers or whatever) then the problem is 1 of 2 things:

1: The splunkd instance has not been restarted on the forwarder. You need to set the Restart Splunk setting for the App that contains outputs.conf inside of your Deployment Server.

2: There is another outputs.conf file on the forwarder that has a higher precedence that the one that you are deploying. This probably means $SPLUNK_HOME/etc/system/local (or $SPLUNK_HOME/etc/system/default, which is just WRONG, but check anyway). Anything in system/local is GOD and cannot be overridden with one exception: slave-apps in an Indexer Cluster.

[aaraneta_splunk]

@avalle - Did one of the answers below help provide a solution your question? If yes, please click “Accept” below the best answer to resolve this post and upvote anything that was helpful. If no, please leave a comment with more feedback. Thanks.

[woodcock · Accepted solution]

The replacement of apps on the forwarder is atomic and this confuses many people. The process is like this:

DC(forwarder) -> DC: "I am host X"
DS            -> DC: "Send me checksums for these apps: A, B, C, D"
DC            -> DS: "A=123, B=234, C=345, D=<null>"
DS            -> DC: "Install this 'D' and replace your 'A' with this"
DC                 : cd $SPLUNK_HOME/etc/apps; mv A /tmp/, tar xvf A.tgz; merge /tmp/A/local with ./A/local

It may not be exactly like this but for the purposes of this question, it proves this point: If you are sitting in the A directory on the forwarder and watching for files to change, you will never see the changes because you are in the old directory that has been trashed. You need to cd out of the old/moved/trashed directory into the new/replacement/merged directory by again doing cd $SPLUNK_HOME/etc/apps/A/ after a few minutes to make sure that you are in the right place to see the changes.

[Richfez]

It depends by what you mean by "does not update the outputs.conf on the clients?"

If you mean the App1/local/outputs.conf doesn't get updated (physically checking the file on clients that show as having checked in recently don't show the update?) then are you sure you ran a splunk reload deploy-server as per the documentation on deploying apps?

If instead the files themselves update (are physically changed) but the settings don't "take effect", then perhaps you've hit a file precedence issue. Apps can't override settings made in $SPLUNK_HOME/etc/system/local/. In that case, confirm your app's files are in place and *should work, then delete the local copy out of the system/local version and confirm that works for a few test cases, then deploy whatever it is you did to delete those files more widely once you are happy..

[avalle]

Thank you so much for your answer! I did reload the deployment server and the configuration file I had in $SPLUNK_HOME/etc/deployment-apps/App1/local disappeared and the Forwarders did not get the updated configuration files therefore I am not ready to delete the conf files in the local directory until I get thus update to happen in the UF.

[avalle]

Apparently I was looking in the wrong UF directory.
The configuration files in the UF have been updated on $SPLUNK_HOME\etc\apps\security\local but for some reason the new entries are all in one line. Do I have to remove the conf files in the etc\system\local directory to test if these new files work?

[Richfez]

The configuration file precedence docs page explains how this works, and I believe this link will take you right to the precedence section.

But, basically, yes. That last link above has a paragraph that's quite clear - if you have a setting set in the etc/system/local/blah.conf file, that setting takes precedence over all others.

As a concrete example, when I deploy my UFs manually (which I do for one-off installs or whatever) I supply a deployment server IP address and port for the UF to contact for its configuration. That writes a etc/system/local/deploymentclient.conf file on the UF system. Splunk starts, contacts the DS named in that file and gets its configuration, right? So far fine.

One of the files in one of the apps I have deployed is etc/apps/mycompany_deploymentclient/local/deploymentclient.conf. THIS deploymentclient.conf is my official one, and if I have to move or change my Deployment Server, I can set up the new DS and then change this file to point to the new server. The UFs will contact the old server, get a copy of the new configuration, restart and start talking to the NEW DS, right?

But NOT if they still have a etc/system/local/deploymentclient.conf. If that latter file exists, it's settings will override anything I could send in an app. So, basically for the one-off installs I use this method for I install the UF with a DS setup, let it start and get its "real" config (which includes the deploymentclient.conf file it really needs) and then delete the system/local/ version and restart it again.

Now that's an example of file-level precedence, but settings inside files work the same way. If in an etc/system/local/X.conf file you have setting Z, then NO OTHER setting for Z elsewhere will ever override the way it's set in etc/system/local/X.conf.

Indeed, this is a great feature once it's understood. For instance, all the default settings are in etc/system/default/blah.conf <- the LOWEST precedence files. That means that all other settings will take effect (according to their own respective precedences) and only if no other file has a particular setting that's set in default will that setting take the default.

Make sense?

[ddrillic]

You mentioned $SPLUNK_HOME/etc/deployment-apps/App1/local/outputs.conf. What about its twin? inputs.conf?

[avalle]

Right now I am working on updating one conf file at a time it does not matter which one as long as i get one working.

[somesoni2]

You need to assign your App1 to the servers who should receive. From your description, you've already completed step "create deployment app" https://docs.splunk.com/Documentation/Splunk/6.5.1/Updating/Createdeploymentapps. If you're going to use Deployment Server Web UI -> Forwarder Management dashboard to configure clients, you need to reload your deployment server now, to make the App1 visible in UI. Now use steps from following link to add/update serverclass, add clients and App1. https://docs.splunk.com/Documentation/Splunk/6.5.1/Updating/Useforwardermanagement

Same from serverclass.conf file directly:- http://docs.splunk.com/Documentation/Splunk/6.5.1/Updating/Useserverclass.conf

[avalle]

I reloaded the deployment server and the configuration files under the $SPLUNK_HOME/etc/deployment-apps/App1/local directory disappeared and the UF did not get the updated configuration files.