How to set up Multiple Deployment Servers Configuration

[rdevine]

we will have lots of splunk forwarders at different sites in our environment and are looking to reduce cross talk between the sites. what would be the best way to setup splunk deployment servers in this instance.

My original thoughts are having one at each site, but how do i keep the configurations synced?
Is there a way for forwarders talk to multiple deployment servers, or better yet, default to a specific deployment server based on location?

Thanks

[Eqalis · Accepted solution]

This can be done. You set up two deployment servers, one in each location. You set one as a client of the other and give it a server class to deploy all the deployment-apps directories to the deployment apps directory of the other.

Give each client a site app which tells it where to deploy from and forward to.

It should look somthing like this

  Site 1 Master                        Site 2 Slave
  Deployment Server                    Deployment Server
Deployment apps dir----------------->Deployment apps dir
         |                                    |
         |                                    |
Client A |                           Client B |
         v                                    v
      apps dir                             apps dir
  ______|______                         ______|______
  |           |                         |           |
Site1 app    Other Apps              Site2 app    Other Apps

The important thing is that the slave deployment server should reload whenever it gets a deployment app from the master. This forces it to deploy the new app to it's clients.

With this setup, you only have one server polling across the wan and only deploy across it once per update.

[AnilPujar]

@Eqalis 

can you please share if there is any Splunk docs link for implementing this type of architecture?

I found one, https://conf.splunk.com/files/2019/slides/FN2048.pdf but its just explanations.

How to setup exactly if splunk has any documentation will be helpful.

Thanks 

Anil

[infodon]

We have had no success in trying to establish this configuration. We have followed similar configuration for tiered migration but the second tier (slave) does not get the apps from the master. Do you have any sample config from both the master and secondary for your success that you could share?

[Eqalis]

I tend to deploy 4 types of app. A site app as above, An OS app (unix,win,mac) for any operating system specific info, A splunk function app (UF, LWF, HWF, Indexer, SearchHead, DeploymentServer) and SOME server function apps defining Inputs etc. This gives maximum control and flexibility without becoming impossible to manage.

It also means I can look at any deployment client in the app directory and see this server is for example in London, running Windows, is a LightWeightForwarder and collects IIS logs and perfmon.

[GauravSplunxter]

I am also stuck in technicalities of this. Can we get some sample configs for this?

[Eqalis]

You create an app that is specific to each site. Lets call them London and Sanfran. In each one you create the files that are specific to that site. I would at a minimum have an outputs.conf pointing to your local indexer and a deploymentclient.conf pointing to your local deployment server.

That way devices in London get the London app, download config from the London DS and send data to the London indexer. Similarly devices in Sanfran get their app and connect to devices in Sanfran.

[rdevine]

What do you mean by a "site app"?

[dwaddle]

You can always do some DNS trickery with CNAMES and such using the default DNS search algorithm. Say you push a deploymentclient.conf that says simply that the deployment server is splunkdeploymentserver. Then, in each site's ${CITY}.company.net DNS zones, put in a CNAME to the proper deployment server for that site. DNS will handle adding the default suffix from resolv.conf, and resolve splunkdeploymentserver to splunkdeploymentserver.${CITY}.company.net. Your CNAMEs then handle the rest.

Then, keep all of your various deployment servers in sync using rsync. Make one the master, and make the rest rsync off it. Or, if you're feeling extra fancy, use git and let them do a pull from a master git source. Built in version control, woo!

[jbsplunk]

Why not just use 2 different deploymentclient.conf files, one for each type of forwarder? You could use puppet to push out your installation with the deployment client and be done with it.

[anand_singh17]

This we tried in our test environment, it did not work out. 😞

[jbsplunk]

Understood. I wasn't thinking that the forwarders were different types. However, you can still deploy 2 sets of deploymentclient.conf files to point to 2 different deployment servers with different apps to be pushed out. This is a very common scenario.

[rdevine]

The difficulty (as i see it, i could be wrong) is that the configuration files would be the same. the difference isn't in the type of forwarders, but in their physical location.