Three tier deployment topology not working right

[mikehodges01]

I have a three-tier tiered deployment environment setup. On my 2 deployment clients that also are deployment servers, I have put reloadDSOnAppInstall = true to deploymentclient.conf. I was getting some errors in my splunkd.log about failing to install the apps on the client/server. I found this thread - http://splunk-base.splunk.com/answers/10500/tiered-deployment-servers-is-it-possible - and abandoned using targetRepositoryLocation and instead specifying the default app "install" directory, /SPLUNK_HOME/etc/apps, as my repositoryLocation for the 2 deployment client/servers. The errors have stopped showing up in splunkd.log but it seems that I am still unable to get a file to move even one step in my deployment chain, unless I go to all three deployment servers and use the ./splunk reload deploy-server command. So the problem is that replication is working, but it appears that the reloadDSOnAppInstall clause in my deploymentclient.conf is doing absolutely nothing. Please help!

[dmesler]

Interesting, so my tiered approach of having separate subdirectories for searchheads, indexers, and forwarders is what's preventing my deployment server from automatically rereading the contents of ../deployment-apps?

I've been having to run 'splunk reload deploy-server' or 'splunk refresh deploy-clients' (either works) to get the apps to deploy and wondered why the docs made no mention of having to do this. Seems like this is a bug that needs squashing, have you submitted a bug report? Since running 'splunk reload deploy-server' requires authentication this has made automation impossible. (Ok, not impossible, but it would mean having to put a username/password in a config or script somewhere, blech.)

[mikehodges01 · Accepted solution]

Ok, I answered my own question.

For anyone else out there wondering about something similar, all folders that contain apps MUST be on the same folder level, and on the initial deployment server, that MUST be /SPLUNK_HOME/etc/deployment-apps. What I had was three folders with the "apps" for my 1st tier (for example, SPLUNKHOME/etc/deployment-apps/App/default/whatev.conf), and then two folders, each with three "apps", for the other two tiers (so, for example, SPLUNKHOME/etc/deployment-apps/Tier2/App/default/whatev.conf). As soon as I moved ALL the folders out of those last two and up one level into /SPLUNKHOME/etc/deployment-apps and changed my serverclass.conf to reflect the new locations, everything started working. So it appears you can have tiered deployment, with automatic replication working thanks to the "reloadDSOnAppInstall = true" parameter as long as you don't ever nest your app directories.

Keep in mind also that the "apps" that need to be deployed to a client/server should be installed ONLY in the default directory, /SPLUNK_HOME/etc/apps, in order for "reloadDSOnAppInstall = true" to function. Use repositoryLocation on the client/servers' serverclass.conf to specify where its clients should get their content. DO NOT change the default installation directory by using targetRepositoryLocation or this won't work. I'm on Splunk 4.3.2, and I'm surprised Splunk hasn't fixed this one yet!