Deployment Architecture

Multiple Deployment Servers Configuration

Path Finder

we will have lots of splunk forwarders at different sites in our environment and are looking to reduce cross talk between the sites. what would be the best way to setup splunk deployment servers in this instance.

My original thoughts are having one at each site, but how do i keep the configurations synced?
Is there a way for forwarders talk to multiple deployment servers, or better yet, default to a specific deployment server based on location?

Thanks

1 Solution

Explorer

This can be done. You set up two deployment servers, one in each location. You set one as a client of the other and give it a server class to deploy all the deployment-apps directories to the deployment apps directory of the other.

Give each client a site app which tells it where to deploy from and forward to.

It should look somthing like this

  Site 1 Master                        Site 2 Slave
  Deployment Server                    Deployment Server
Deployment apps dir----------------->Deployment apps dir
         |                                    |
         |                                    |
Client A |                           Client B |
         v                                    v
      apps dir                             apps dir
  ______|______                         ______|______
  |           |                         |           |
Site1 app    Other Apps              Site2 app    Other Apps

The important thing is that the slave deployment server should reload whenever it gets a deployment app from the master. This forces it to deploy the new app to it's clients.

With this setup, you only have one server polling across the wan and only deploy across it once per update.

View solution in original post

Explorer

This can be done. You set up two deployment servers, one in each location. You set one as a client of the other and give it a server class to deploy all the deployment-apps directories to the deployment apps directory of the other.

Give each client a site app which tells it where to deploy from and forward to.

It should look somthing like this

  Site 1 Master                        Site 2 Slave
  Deployment Server                    Deployment Server
Deployment apps dir----------------->Deployment apps dir
         |                                    |
         |                                    |
Client A |                           Client B |
         v                                    v
      apps dir                             apps dir
  ______|______                         ______|______
  |           |                         |           |
Site1 app    Other Apps              Site2 app    Other Apps

The important thing is that the slave deployment server should reload whenever it gets a deployment app from the master. This forces it to deploy the new app to it's clients.

With this setup, you only have one server polling across the wan and only deploy across it once per update.

View solution in original post

Explorer

We have had no success in trying to establish this configuration. We have followed similar configuration for tiered migration but the second tier (slave) does not get the apps from the master. Do you have any sample config from both the master and secondary for your success that you could share?

Explorer

I tend to deploy 4 types of app. A site app as above, An OS app (unix,win,mac) for any operating system specific info, A splunk function app (UF, LWF, HWF, Indexer, SearchHead, DeploymentServer) and SOME server function apps defining Inputs etc. This gives maximum control and flexibility without becoming impossible to manage.

It also means I can look at any deployment client in the app directory and see this server is for example in London, running Windows, is a LightWeightForwarder and collects IIS logs and perfmon.

0 Karma

I am also stuck in technicalities of this. Can we get some sample configs for this?

0 Karma

Explorer

You create an app that is specific to each site. Lets call them London and Sanfran. In each one you create the files that are specific to that site. I would at a minimum have an outputs.conf pointing to your local indexer and a deploymentclient.conf pointing to your local deployment server.

That way devices in London get the London app, download config from the London DS and send data to the London indexer. Similarly devices in Sanfran get their app and connect to devices in Sanfran.

0 Karma

Path Finder

What do you mean by a "site app"?

0 Karma

SplunkTrust
SplunkTrust

You can always do some DNS trickery with CNAMES and such using the default DNS search algorithm. Say you push a deploymentclient.conf that says simply that the deployment server is splunkdeploymentserver. Then, in each site's ${CITY}.company.net DNS zones, put in a CNAME to the proper deployment server for that site. DNS will handle adding the default suffix from resolv.conf, and resolve splunkdeploymentserver to splunkdeploymentserver.${CITY}.company.net. Your CNAMEs then handle the rest.

Then, keep all of your various deployment servers in sync using rsync. Make one the master, and make the rest rsync off it. Or, if you're feeling extra fancy, use git and let them do a pull from a master git source. Built in version control, woo!

Splunk Employee
Splunk Employee

Why not just use 2 different deploymentclient.conf files, one for each type of forwarder? You could use puppet to push out your installation with the deployment client and be done with it.

Path Finder

This we tried in our test environment, it did not work out. 😞

0 Karma

Splunk Employee
Splunk Employee

Understood. I wasn't thinking that the forwarders were different types. However, you can still deploy 2 sets of deploymentclient.conf files to point to 2 different deployment servers with different apps to be pushed out. This is a very common scenario.

0 Karma

Path Finder

The difficulty (as i see it, i could be wrong) is that the configuration files would be the same. the difference isn't in the type of forwarders, but in their physical location.

0 Karma