Tiered Deployment Servers, is it possible?

[pl123]

Hi, We are working on planning for a large splunk deployment, I would like to know if it is all possible to set up a deployment server, which pushes out configurations / apps to another deployment server, which in turn pushes these out to lightfowarders & indexers on the other side of a firewall (from the original deployment server).

Also is it possible for an instance to be both a light forwarder and a deployment server?

Cheers

[steven_swor]

Here's how I did it late last year. It works really well (I also have a better-documented version of this example at https://gist.github.com/sworisbreathing/d236895568eaf31da579).
In this example, we're focusing mostly on the TAs for SOS as well as an arbitrary app.

On the primary deployment server (e.g. deployment master), in serverclass.conf:

###########################################################
# Secondary deployment servers - Deployment Apps
###########################################################
[serverClass:secondaryDeploymentServersDeploymentApps]
targetRepositoryLocation = $SPLUNK_HOME/etc/deployment-apps
whitelist.0 = *SecondaryDeploymentServer*
stateOnClient = noop

[serverClass:secondaryDeploymentServersDeploymentApps:app:TA-sos_win]
[serverClass:secondaryDeploymentServersDeploymentApps:app:TA-sos]

###########################################################
# Secondary deployment servers - Self-Deployed Apps
###########################################################
[serverClass:secondaryDeploymentServers]
whitelist.0 = *SecondaryDeploymentServer*

[serverClass:secondaryDeploymentServers:app:foo]
restartSplunkd = true


###########################################################
# Linux/Unix Splunk Instances
# (note: this is probably not a comprehensive list)
###########################################################
[serverClass:NixHosts]
whitelist.0 = *
machineTypesFilter = linux-*, freebsd-*, darwin-*, sunos-*, aix-*

[serverClass:NixHosts:app:TA-sos]
restartSplunkd = true

###########################################################
# Windows Splunk Instances
###########################################################
[serverClass:WindowsHosts]
whitelist.0 = *
machineTypesFilter = windows-*

[serverClass:NixHosts:app:TA-sos_win]
restartSplunkd = true

On the secondary deployment server (e.g. deployment slave), in deploymentclient.conf:

[deployment-client]
clientName = SecondaryDeploymentServer
reloadDSOnAppInstall = true

[target-broker:PrimaryDeploymentServer]
targetUri = <deployment_master_fqdn>:<deployment_master_mgmt_port>

On the secondary deployment server (e.g. deployment slave), in serverclass.conf:

###########################################################
# Linux/Unix Splunk Instances
# (note: this is probably not a comprehensive list)
###########################################################
[serverClass:NixHosts]
whitelist.0 = *
machineTypesFilter = linux-*, freebsd-*, darwin-*, sunos-*, aix-*

[serverClass:NixHosts:app:TA-sos]
restartSplunkd = true

###########################################################
# Windows Splunk Instances
###########################################################
[serverClass:WindowsHosts]
whitelist.0 = *
machineTypesFilter = windows-*

[serverClass:NixHosts:app:TA-sos_win]
restartSplunkd = true

The big thing to call out here is that any time you're deploying to somewhere other than $SPLUNK_HOME/etc/apps, you need to specify stateOnClient = noop in serverclass.conf. Otherwise, you'll see a bunch of errors on the deployment client about not being able to properly install the app (and if you also specified restartSplunkd = true in serverclass.conf, the deployment client will just sit there and restart the Splunk service over and over again)

[TonyLeeVT]

Thanks for the post sir. Just an FYI, your hyperlink to github includes your close parenthesis and period.

For example: https://gist.github.com/sworisbreathing/d236895568eaf31da579%29.

The fix is to remove '%29.'

Thanks again

[jasonstone]

At what version will this be fixed? It makes a tiered deployment pretty much useless. I would like to be able to manage my secondary deployment servers from a single master...client apps and it's own local app.

[gr]

Wow. Two major versions down the drain, and this still isn't fixed?

I'm glad that my hacky answer above helped some people, but this is absurd.

[cmeo]

Update for 2014

We are currently investigating this feature. Tiered deployment servers IS possible, but there are three serious limitations which prevent it from being truly useful. Enhancements are needed.

1. targetRepositoryLocation has to be set at the [global] level. This means that if you have a deploy server in a subnet, it only knows how to deploy to $SPLUNK_HOME/etc/apps or somewhere else e.g. deployment-apps, but not both. You need to be able to override this at the server class level, so a single deploy server can feed both intermediate deployment servers, and clients in its network segment if you need it to. At the moment, if you need to deploy to intermediate servers and to clients in the same subnet, you need two deploy servers (at least), with different values for targetRepositoryLocation set. You want the app present but disabled in deployment-apps on the second tier servers, and present but enabled in apps on clients. You may also need to deploy the app to run on the second tier servers as well--impossible as it sits, unless you use two deployment servers.
2. Having set targetDeploymentRepository, your web UI gets turned off and you have to do everything else via CLI or editing the files. Painful-and unnecessary?
3. Client status is not currently forwarded from the intermediate servers to a master deploy server. This means you need a heavy server in each subnet just to view forwarder status, and you don't have a 'global' view anywhere. There needs to be an option to send the client status to a master deploy server which can see everything. This is probably the most serious issue.

Our customer will not allow deploy clients to talk directly to a deploy server from firewalled subnets so that's not an option.

With these limitations we've decided not to use tiered deployment as it doesn't add much value. Instead we will have a deployment server in each subnet, and update them separately, which is far from ideal.

It would also be nice if the deployment server could deploy to itself. I don't think this is possible at the moment.

Good feature, but in need of improvement.

[lmyrefelt]

Hi cmeo, take a look at the last section of; http://wiki.splunk.com/Deploy:DeploymentServer , to see if this could help you ?

Here they are overwriting the location for where the clients are downloading (on the "sub-deployment-server" ) the apps, from the client side, which seems to be able to give you possibilities to admin the "sub-deployment-clients" from the GUI (on the sub-deployment-server) if thats what you want ?

[infodon]

We have tried to unsuccessfully to implement this configuration. We are pretty sure we have everything setup right but the tier2 deployment server never gets any apps in either the "apps" or "deployment-apps" folders?? Ideas?

[jaxjohnny2000]

https://gist.github.com/sworisbreathing/d236895568eaf31da579 will replicate the apps. However, you need an rsync or cron job to copy the serverclass.conf over.

[gr · Accepted solution]

Actually, the behavior on the deployment client side to the use of targetRepositoryLocation appears to be buggy: if you use it, the second tier deployment server (also a deployment client of the first tier deployment server) will forever try, and fail, to "install" the apps pushed to the alternate targetRepositoryLocation (say, $SPLUNK_HOME/etc/deployment-apps), but fail to do so. If you've also told it, in deploymentclient.conf, to restart deployment server when new apps are installed (which you almost definitely need to do), then that'll happen every pollingInterval, and your final deployment clients will keep getting told to redo their handshakes. The repeated attempts and failures to install the apps on the second-tier deployment server take place regardless of whether the server class on the first-tier deployment server has "stateOnClient" set to enabled or disabled.

I just tested all of this with 4.2.2 on RHEL Linux 5.6 systems, and the work-around I'm using for multi-tier deployment is to ignore targetRepositoryLocation and, instead, tell serverclass.conf on the second-tier deployment server that its repository is $SPLUNK_HOME/etc/apps, having set the apps I want that server to deploy (but not run) to be disabled. The details on how that works follow.

On the primary deployment server, deploy-svr-1, I've got two classes for my slave deployment server, deploy-svr-2, in serverclass.conf. They look like this:

[serverClass:Tier2-DeploymentServer]
whitelist.0=deploy-svr-2
[serverClass:Tier2-DeploymentServer:app:Tier2-DeploymentServer]

[serverClass:Tier2-DeploymentApps]
whitelist.0=deploy-svr-2
stateOnClient = disabled
[serverClass:Tier2-DeploymentApps:app:Tier2-Inputs]
[serverClass:Tier2-DeploymentApps:app:Tier2-Outputs]
[serverClass:Tier2-DeploymentApps:app:DeploymentClient]
[... so forth; anything that goes on the final deployment clients]

Then we tell deploy-svr-2 to be a deployment client of deploy-svr-1, reloading its own deployment server configuration after it's installed new apps (so that it will then push any of those out to the lower tier that it needs to):

[deployment-client]
reloadDSOnAppInstall = true

[target-broker:deploymentServer]
targetUri = deploy-svr-1:8089

The Tier2-DeploymentServer app on deploy-svr-1 contains only its own serverclass.conf for the final deployment clients, which looks like this:

[global]
continueMatching = true
restartSplunkd = true
repositoryLocation = $SPLUNK_HOME/etc/apps

[serverClass:Tier2-Forwarders]
whitelist.0=*
stateOnClient = enabled
[serverClass:Tier2-Forwarders:app:Tier2-Inputs]
[serverClass:Tier2-Forwarders:app:Tier2-Outputs]
[serverClass:Tier2-Forwarders:app:DeploymentClient]

As you can see, I'm telling deploy-svr-2's deployment server to look for apps to deploy right in $SPLUNK_HOME/etc/apps, where deploy-svr-1 will have pushed them, and set them to be disabled. (Setting stateOnClient explicitly in the Tier2-Forwarders server class on deploy-svr-2 is actually redundant--that's the default--but I've got it in there for clarity.)

Finally, the deploymentclient.conf for final tier deployment clients looks like this:

[deployment-client]

[target-broker:deploymentServer]
targetUri = deploy-svr-2:8089

The one caveat here is that we have to avoid having any of the apps that deploy-svr-2 will deploy also need to run on deploy-svr-2 itself, which will typically be inputs.conf (to pick up local log files and potentially run scripted inputs) and outputs.conf (pointing back to the main Splunk indexing cluster). In my case, that's fine, because deploy-svr-2 is also intended to be a forwarder for logs from systems within a separate network, so I need to have a different configuration for deployment clients in that environment (sending their events to deploy-svr-2) than I will for deploy-svr-2 itself (sending events to my main indexing cluster), but it's important to remember that if you want an app to run on deploy-svr-2, you can't include it in deploy-svr-1's Tier2-DeploymentApps server class, because that will get it disabled on deploy-svr-2.

Incidentally, the deployment server class documentation that Genti pointed to above has since moved to here.

[infodon]

We have tried to unsuccessfully to implement this configuration. We are pretty sure we have everything setup right but the tier2 deployment server never gets any apps in either the "apps" or "deployment-apps" folders?? Ideas?

[Genti]

Yes, that configuration is possible, however it is a bit tedious work and should be setup by a splunk expert as it is prone to a lot of mistakes.

The important thing here is this attribute:

targetRepositoryLocation

Please check this page for more info.

The idea is you need to setup the 1st level deployment server to send the data to the "/deployment-apps/" directory instead of the /apps/.

The answer to your second question is : YES.

[Jason]

This is true in theory, but a bug means you can't do this the logical way. As of 4.1.8 (likely earlier) - DO NOT use Deployment Server to install into anything other than a client's etc/apps/ directory. You will get unpredictiable results and unstable end forwarders. See below. You need to install the slave's clients' apps into the slave's etc/apps/, but as disabled. Then use etc/apps/ as repositoryLocation.