Deployment Architecture

How to set up Multiple Deployment Servers Configuration

rdevine
Path Finder

we will have lots of splunk forwarders at different sites in our environment and are looking to reduce cross talk between the sites. what would be the best way to setup splunk deployment servers in this instance.

My original thoughts are having one at each site, but how do i keep the configurations synced?
Is there a way for forwarders talk to multiple deployment servers, or better yet, default to a specific deployment server based on location?

Thanks

Labels (1)
1 Solution

Eqalis
Explorer

This can be done. You set up two deployment servers, one in each location. You set one as a client of the other and give it a server class to deploy all the deployment-apps directories to the deployment apps directory of the other.

Give each client a site app which tells it where to deploy from and forward to.

It should look somthing like this

  Site 1 Master                        Site 2 Slave
  Deployment Server                    Deployment Server
Deployment apps dir----------------->Deployment apps dir
         |                                    |
         |                                    |
Client A |                           Client B |
         v                                    v
      apps dir                             apps dir
  ______|______                         ______|______
  |           |                         |           |
Site1 app    Other Apps              Site2 app    Other Apps

The important thing is that the slave deployment server should reload whenever it gets a deployment app from the master. This forces it to deploy the new app to it's clients.

With this setup, you only have one server polling across the wan and only deploy across it once per update.

View solution in original post

Eqalis
Explorer

This can be done. You set up two deployment servers, one in each location. You set one as a client of the other and give it a server class to deploy all the deployment-apps directories to the deployment apps directory of the other.

Give each client a site app which tells it where to deploy from and forward to.

It should look somthing like this

  Site 1 Master                        Site 2 Slave
  Deployment Server                    Deployment Server
Deployment apps dir----------------->Deployment apps dir
         |                                    |
         |                                    |
Client A |                           Client B |
         v                                    v
      apps dir                             apps dir
  ______|______                         ______|______
  |           |                         |           |
Site1 app    Other Apps              Site2 app    Other Apps

The important thing is that the slave deployment server should reload whenever it gets a deployment app from the master. This forces it to deploy the new app to it's clients.

With this setup, you only have one server polling across the wan and only deploy across it once per update.

AnilPujar
Path Finder

@Eqalis 

can you please share if there is any Splunk docs link for implementing this type of architecture?

I found one, https://conf.splunk.com/files/2019/slides/FN2048.pdf but its just explanations.

How to setup exactly if splunk has any documentation will be helpful.

Thanks 

Anil

 

 

0 Karma

infodon
Explorer

We have had no success in trying to establish this configuration. We have followed similar configuration for tiered migration but the second tier (slave) does not get the apps from the master. Do you have any sample config from both the master and secondary for your success that you could share?

Eqalis
Explorer

I tend to deploy 4 types of app. A site app as above, An OS app (unix,win,mac) for any operating system specific info, A splunk function app (UF, LWF, HWF, Indexer, SearchHead, DeploymentServer) and SOME server function apps defining Inputs etc. This gives maximum control and flexibility without becoming impossible to manage.

It also means I can look at any deployment client in the app directory and see this server is for example in London, running Windows, is a LightWeightForwarder and collects IIS logs and perfmon.

0 Karma

GauravSplunxter
Explorer

I am also stuck in technicalities of this. Can we get some sample configs for this?

0 Karma

Eqalis
Explorer

You create an app that is specific to each site. Lets call them London and Sanfran. In each one you create the files that are specific to that site. I would at a minimum have an outputs.conf pointing to your local indexer and a deploymentclient.conf pointing to your local deployment server.

That way devices in London get the London app, download config from the London DS and send data to the London indexer. Similarly devices in Sanfran get their app and connect to devices in Sanfran.

0 Karma

rdevine
Path Finder

What do you mean by a "site app"?

0 Karma

dwaddle
SplunkTrust
SplunkTrust

You can always do some DNS trickery with CNAMES and such using the default DNS search algorithm. Say you push a deploymentclient.conf that says simply that the deployment server is splunkdeploymentserver. Then, in each site's ${CITY}.company.net DNS zones, put in a CNAME to the proper deployment server for that site. DNS will handle adding the default suffix from resolv.conf, and resolve splunkdeploymentserver to splunkdeploymentserver.${CITY}.company.net. Your CNAMEs then handle the rest.

Then, keep all of your various deployment servers in sync using rsync. Make one the master, and make the rest rsync off it. Or, if you're feeling extra fancy, use git and let them do a pull from a master git source. Built in version control, woo!

jbsplunk
Splunk Employee
Splunk Employee

Why not just use 2 different deploymentclient.conf files, one for each type of forwarder? You could use puppet to push out your installation with the deployment client and be done with it.

anand_singh17
Path Finder

This we tried in our test environment, it did not work out. 😞

0 Karma

jbsplunk
Splunk Employee
Splunk Employee

Understood. I wasn't thinking that the forwarders were different types. However, you can still deploy 2 sets of deploymentclient.conf files to point to 2 different deployment servers with different apps to be pushed out. This is a very common scenario.

0 Karma

rdevine
Path Finder

The difficulty (as i see it, i could be wrong) is that the configuration files would be the same. the difference isn't in the type of forwarders, but in their physical location.

0 Karma
Get Updates on the Splunk Community!

Introduction to Splunk Observability Cloud - Building a Resilient Hybrid Cloud

Introduction to Splunk Observability Cloud - Building a Resilient Hybrid Cloud  In today’s fast-paced digital ...

Observability protocols to know about

Observability protocols define the specifications or formats for collecting, encoding, transporting, and ...

Take Your Breath Away with Splunk Risk-Based Alerting (RBA)

WATCH NOW!The Splunk Guide to Risk-Based Alerting is here to empower your SOC like never before. Join Haylee ...