Deployment Architecture

Tiered Deployment Servers, is it possible?

pl123
Path Finder

Hi, We are working on planning for a large splunk deployment, I would like to know if it is all possible to set up a deployment server, which pushes out configurations / apps to another deployment server, which in turn pushes these out to lightfowarders & indexers on the other side of a firewall (from the original deployment server).

Also is it possible for an instance to be both a light forwarder and a deployment server?

Cheers

Tags (2)
1 Solution

gr
Explorer

Actually, the behavior on the deployment client side to the use of targetRepositoryLocation appears to be buggy: if you use it, the second tier deployment server (also a deployment client of the first tier deployment server) will forever try, and fail, to "install" the apps pushed to the alternate targetRepositoryLocation (say, $SPLUNK_HOME/etc/deployment-apps), but fail to do so. If you've also told it, in deploymentclient.conf, to restart deployment server when new apps are installed (which you almost definitely need to do), then that'll happen every pollingInterval, and your final deployment clients will keep getting told to redo their handshakes. The repeated attempts and failures to install the apps on the second-tier deployment server take place regardless of whether the server class on the first-tier deployment server has "stateOnClient" set to enabled or disabled.

I just tested all of this with 4.2.2 on RHEL Linux 5.6 systems, and the work-around I'm using for multi-tier deployment is to ignore targetRepositoryLocation and, instead, tell serverclass.conf on the second-tier deployment server that its repository is $SPLUNK_HOME/etc/apps, having set the apps I want that server to deploy (but not run) to be disabled. The details on how that works follow.

On the primary deployment server, deploy-svr-1, I've got two classes for my slave deployment server, deploy-svr-2, in serverclass.conf. They look like this:

[serverClass:Tier2-DeploymentServer]
whitelist.0=deploy-svr-2
[serverClass:Tier2-DeploymentServer:app:Tier2-DeploymentServer]

[serverClass:Tier2-DeploymentApps]
whitelist.0=deploy-svr-2
stateOnClient = disabled
[serverClass:Tier2-DeploymentApps:app:Tier2-Inputs]
[serverClass:Tier2-DeploymentApps:app:Tier2-Outputs]
[serverClass:Tier2-DeploymentApps:app:DeploymentClient]
[... so forth; anything that goes on the final deployment clients]

Then we tell deploy-svr-2 to be a deployment client of deploy-svr-1, reloading its own deployment server configuration after it's installed new apps (so that it will then push any of those out to the lower tier that it needs to):

[deployment-client]
reloadDSOnAppInstall = true

[target-broker:deploymentServer]
targetUri = deploy-svr-1:8089

The Tier2-DeploymentServer app on deploy-svr-1 contains only its own serverclass.conf for the final deployment clients, which looks like this:

[global]
continueMatching = true
restartSplunkd = true
repositoryLocation = $SPLUNK_HOME/etc/apps

[serverClass:Tier2-Forwarders]
whitelist.0=*
stateOnClient = enabled
[serverClass:Tier2-Forwarders:app:Tier2-Inputs]
[serverClass:Tier2-Forwarders:app:Tier2-Outputs]
[serverClass:Tier2-Forwarders:app:DeploymentClient]

As you can see, I'm telling deploy-svr-2's deployment server to look for apps to deploy right in $SPLUNK_HOME/etc/apps, where deploy-svr-1 will have pushed them, and set them to be disabled. (Setting stateOnClient explicitly in the Tier2-Forwarders server class on deploy-svr-2 is actually redundant--that's the default--but I've got it in there for clarity.)

Finally, the deploymentclient.conf for final tier deployment clients looks like this:

[deployment-client]

[target-broker:deploymentServer]
targetUri = deploy-svr-2:8089

The one caveat here is that we have to avoid having any of the apps that deploy-svr-2 will deploy also need to run on deploy-svr-2 itself, which will typically be inputs.conf (to pick up local log files and potentially run scripted inputs) and outputs.conf (pointing back to the main Splunk indexing cluster). In my case, that's fine, because deploy-svr-2 is also intended to be a forwarder for logs from systems within a separate network, so I need to have a different configuration for deployment clients in that environment (sending their events to deploy-svr-2) than I will for deploy-svr-2 itself (sending events to my main indexing cluster), but it's important to remember that if you want an app to run on deploy-svr-2, you can't include it in deploy-svr-1's Tier2-DeploymentApps server class, because that will get it disabled on deploy-svr-2.

Incidentally, the deployment server class documentation that Genti pointed to above has since moved to here.

View solution in original post

steven_swor
Path Finder

Here's how I did it late last year. It works really well (I also have a better-documented version of this example at https://gist.github.com/sworisbreathing/d236895568eaf31da579).
In this example, we're focusing mostly on the TAs for SOS as well as an arbitrary app.

On the primary deployment server (e.g. deployment master), in serverclass.conf:

###########################################################
# Secondary deployment servers - Deployment Apps
###########################################################
[serverClass:secondaryDeploymentServersDeploymentApps]
targetRepositoryLocation = $SPLUNK_HOME/etc/deployment-apps
whitelist.0 = *SecondaryDeploymentServer*
stateOnClient = noop

[serverClass:secondaryDeploymentServersDeploymentApps:app:TA-sos_win]
[serverClass:secondaryDeploymentServersDeploymentApps:app:TA-sos]

###########################################################
# Secondary deployment servers - Self-Deployed Apps
###########################################################
[serverClass:secondaryDeploymentServers]
whitelist.0 = *SecondaryDeploymentServer*

[serverClass:secondaryDeploymentServers:app:foo]
restartSplunkd = true


###########################################################
# Linux/Unix Splunk Instances
# (note: this is probably not a comprehensive list)
###########################################################
[serverClass:NixHosts]
whitelist.0 = *
machineTypesFilter = linux-*, freebsd-*, darwin-*, sunos-*, aix-*

[serverClass:NixHosts:app:TA-sos]
restartSplunkd = true

###########################################################
# Windows Splunk Instances
###########################################################
[serverClass:WindowsHosts]
whitelist.0 = *
machineTypesFilter = windows-*

[serverClass:NixHosts:app:TA-sos_win]
restartSplunkd = true

On the secondary deployment server (e.g. deployment slave), in deploymentclient.conf:

[deployment-client]
clientName = SecondaryDeploymentServer
reloadDSOnAppInstall = true

[target-broker:PrimaryDeploymentServer]
targetUri = <deployment_master_fqdn>:<deployment_master_mgmt_port>

On the secondary deployment server (e.g. deployment slave), in serverclass.conf:

###########################################################
# Linux/Unix Splunk Instances
# (note: this is probably not a comprehensive list)
###########################################################
[serverClass:NixHosts]
whitelist.0 = *
machineTypesFilter = linux-*, freebsd-*, darwin-*, sunos-*, aix-*

[serverClass:NixHosts:app:TA-sos]
restartSplunkd = true

###########################################################
# Windows Splunk Instances
###########################################################
[serverClass:WindowsHosts]
whitelist.0 = *
machineTypesFilter = windows-*

[serverClass:NixHosts:app:TA-sos_win]
restartSplunkd = true

The big thing to call out here is that any time you're deploying to somewhere other than $SPLUNK_HOME/etc/apps, you need to specify stateOnClient = noop in serverclass.conf. Otherwise, you'll see a bunch of errors on the deployment client about not being able to properly install the app (and if you also specified restartSplunkd = true in serverclass.conf, the deployment client will just sit there and restart the Splunk service over and over again)

TonyLeeVT
Builder

Thanks for the post sir. Just an FYI, your hyperlink to github includes your close parenthesis and period.

For example: https://gist.github.com/sworisbreathing/d236895568eaf31da579%29.

The fix is to remove '%29.'

Thanks again

0 Karma

jasonstone
Explorer

At what version will this be fixed? It makes a tiered deployment pretty much useless. I would like to be able to manage my secondary deployment servers from a single master...client apps and it's own local app.

gr
Explorer

Wow. Two major versions down the drain, and this still isn't fixed?

I'm glad that my hacky answer above helped some people, but this is absurd.

cmeo
Contributor

Update for 2014

We are currently investigating this feature. Tiered deployment servers IS possible, but there are three serious limitations which prevent it from being truly useful. Enhancements are needed.

  1. targetRepositoryLocation has to be set at the [global] level. This means that if you have a deploy server in a subnet, it only knows how to deploy to $SPLUNK_HOME/etc/apps or somewhere else e.g. deployment-apps, but not both. You need to be able to override this at the server class level, so a single deploy server can feed both intermediate deployment servers, and clients in its network segment if you need it to. At the moment, if you need to deploy to intermediate servers and to clients in the same subnet, you need two deploy servers (at least), with different values for targetRepositoryLocation set. You want the app present but disabled in deployment-apps on the second tier servers, and present but enabled in apps on clients. You may also need to deploy the app to run on the second tier servers as well--impossible as it sits, unless you use two deployment servers.
  2. Having set targetDeploymentRepository, your web UI gets turned off and you have to do everything else via CLI or editing the files. Painful-and unnecessary?
  3. Client status is not currently forwarded from the intermediate servers to a master deploy server. This means you need a heavy server in each subnet just to view forwarder status, and you don't have a 'global' view anywhere. There needs to be an option to send the client status to a master deploy server which can see everything. This is probably the most serious issue.

Our customer will not allow deploy clients to talk directly to a deploy server from firewalled subnets so that's not an option.

With these limitations we've decided not to use tiered deployment as it doesn't add much value. Instead we will have a deployment server in each subnet, and update them separately, which is far from ideal.

It would also be nice if the deployment server could deploy to itself. I don't think this is possible at the moment.

Good feature, but in need of improvement.

lmyrefelt
Builder

Hi cmeo, take a look at the last section of; http://wiki.splunk.com/Deploy:DeploymentServer , to see if this could help you ?

Here they are overwriting the location for where the clients are downloading (on the "sub-deployment-server" ) the apps, from the client side, which seems to be able to give you possibilities to admin the "sub-deployment-clients" from the GUI (on the sub-deployment-server) if thats what you want ?

0 Karma

infodon
Explorer

We have tried to unsuccessfully to implement this configuration. We are pretty sure we have everything setup right but the tier2 deployment server never gets any apps in either the "apps" or "deployment-apps" folders?? Ideas?

0 Karma

jaxjohnny2000
Builder

https://gist.github.com/sworisbreathing/d236895568eaf31da579 will replicate the apps. However, you need an rsync or cron job to copy the serverclass.conf over.

0 Karma

gr
Explorer

Actually, the behavior on the deployment client side to the use of targetRepositoryLocation appears to be buggy: if you use it, the second tier deployment server (also a deployment client of the first tier deployment server) will forever try, and fail, to "install" the apps pushed to the alternate targetRepositoryLocation (say, $SPLUNK_HOME/etc/deployment-apps), but fail to do so. If you've also told it, in deploymentclient.conf, to restart deployment server when new apps are installed (which you almost definitely need to do), then that'll happen every pollingInterval, and your final deployment clients will keep getting told to redo their handshakes. The repeated attempts and failures to install the apps on the second-tier deployment server take place regardless of whether the server class on the first-tier deployment server has "stateOnClient" set to enabled or disabled.

I just tested all of this with 4.2.2 on RHEL Linux 5.6 systems, and the work-around I'm using for multi-tier deployment is to ignore targetRepositoryLocation and, instead, tell serverclass.conf on the second-tier deployment server that its repository is $SPLUNK_HOME/etc/apps, having set the apps I want that server to deploy (but not run) to be disabled. The details on how that works follow.

On the primary deployment server, deploy-svr-1, I've got two classes for my slave deployment server, deploy-svr-2, in serverclass.conf. They look like this:

[serverClass:Tier2-DeploymentServer]
whitelist.0=deploy-svr-2
[serverClass:Tier2-DeploymentServer:app:Tier2-DeploymentServer]

[serverClass:Tier2-DeploymentApps]
whitelist.0=deploy-svr-2
stateOnClient = disabled
[serverClass:Tier2-DeploymentApps:app:Tier2-Inputs]
[serverClass:Tier2-DeploymentApps:app:Tier2-Outputs]
[serverClass:Tier2-DeploymentApps:app:DeploymentClient]
[... so forth; anything that goes on the final deployment clients]

Then we tell deploy-svr-2 to be a deployment client of deploy-svr-1, reloading its own deployment server configuration after it's installed new apps (so that it will then push any of those out to the lower tier that it needs to):

[deployment-client]
reloadDSOnAppInstall = true

[target-broker:deploymentServer]
targetUri = deploy-svr-1:8089

The Tier2-DeploymentServer app on deploy-svr-1 contains only its own serverclass.conf for the final deployment clients, which looks like this:

[global]
continueMatching = true
restartSplunkd = true
repositoryLocation = $SPLUNK_HOME/etc/apps

[serverClass:Tier2-Forwarders]
whitelist.0=*
stateOnClient = enabled
[serverClass:Tier2-Forwarders:app:Tier2-Inputs]
[serverClass:Tier2-Forwarders:app:Tier2-Outputs]
[serverClass:Tier2-Forwarders:app:DeploymentClient]

As you can see, I'm telling deploy-svr-2's deployment server to look for apps to deploy right in $SPLUNK_HOME/etc/apps, where deploy-svr-1 will have pushed them, and set them to be disabled. (Setting stateOnClient explicitly in the Tier2-Forwarders server class on deploy-svr-2 is actually redundant--that's the default--but I've got it in there for clarity.)

Finally, the deploymentclient.conf for final tier deployment clients looks like this:

[deployment-client]

[target-broker:deploymentServer]
targetUri = deploy-svr-2:8089

The one caveat here is that we have to avoid having any of the apps that deploy-svr-2 will deploy also need to run on deploy-svr-2 itself, which will typically be inputs.conf (to pick up local log files and potentially run scripted inputs) and outputs.conf (pointing back to the main Splunk indexing cluster). In my case, that's fine, because deploy-svr-2 is also intended to be a forwarder for logs from systems within a separate network, so I need to have a different configuration for deployment clients in that environment (sending their events to deploy-svr-2) than I will for deploy-svr-2 itself (sending events to my main indexing cluster), but it's important to remember that if you want an app to run on deploy-svr-2, you can't include it in deploy-svr-1's Tier2-DeploymentApps server class, because that will get it disabled on deploy-svr-2.

Incidentally, the deployment server class documentation that Genti pointed to above has since moved to here.

infodon
Explorer

We have tried to unsuccessfully to implement this configuration. We are pretty sure we have everything setup right but the tier2 deployment server never gets any apps in either the "apps" or "deployment-apps" folders?? Ideas?

0 Karma

Genti
Splunk Employee
Splunk Employee

Yes, that configuration is possible, however it is a bit tedious work and should be setup by a splunk expert as it is prone to a lot of mistakes.

The important thing here is this attribute:

targetRepositoryLocation

Please check this page for more info.

The idea is you need to setup the 1st level deployment server to send the data to the "/deployment-apps/" directory instead of the /apps/.

The answer to your second question is : YES.

Jason
Motivator

This is true in theory, but a bug means you can't do this the logical way. As of 4.1.8 (likely earlier) - DO NOT use Deployment Server to install into anything other than a client's etc/apps/ directory. You will get unpredictiable results and unstable end forwarders. See below. You need to install the slave's clients' apps into the slave's etc/apps/, but as disabled. Then use etc/apps/ as repositoryLocation.

Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...