Deployment Server: deploying in baby steps

[krussell101]

I have a very simple architecture.

My universal forwarders: A dozen linux hosts and 2 windows hosts.

One splunk host that is indexer, search head and soon-to-be deployment server.

I want to configure the deployment server in baby steps so I don't break something in the process.

I'd like to :

1) Configure the deployment server on my splunk host (create serverclass.conf, and all the etc/deployment-apps directories with requisite content). Stop and re-start splunk.

2) Have current forwarding and indexing continue to work.

3) Configure a deployment client on a single host. Validate that it is working correctly (and so is everything else).

4) Then continue to configure the deployment clients one forwarder at a time.

Is this possible? If I do step #1, will everything continue to work in my environment or is it an all or nothing proposition?

Thanks!

[Brandon_ganem1 · Accepted solution]

1) Your other splunk instances will not automatically hook up to the deployment server, you will need to point them to it. (http://docs.splunk.com/Documentation/Splunk/latest/Deploy/ConfigureDeploymentClients)

What this means is, regardless of the changes made on the deployment server, nothing takes affect on any other splunk instances.

Secondly, your deployment configuration should be configured in stages. You specify what hosts you would like to apply the configuration to in the serverclass.conf. (http://docs.splunk.com/Documentation/Splunk/latest/Admin/Serverclassconf)

Take note of the machine whitelist and blacklists. This is a useful way to configure indexers to receive some apps, search heads others. So on and so forth. There are multiple ways to do this, it is worth spending the time to figure out what works for your deployment.

2) unless you specify a change in the deployment client, nothing is going to change by simply setting up a deployment server. Also note splunk's order of precedence. (http://docs.splunk.com/Documentation/Splunk/latest/admin/Wheretofindtheconfigurationfiles)

(Least precedence) System default < app default < app local < System local (Greatest precedence) (http://docs.splunk.com/Documentation/Splunk/latest/admin/Wheretofindtheconfigurationfiles#Example_of...)

3) its always a great idea to test things before implementing them. Test a forwarder, search head and indexer if feasible.

4) This would be accomplished by hooking them up one by one and applying proper whitelisting and blacklisting.
For example - Have a forwarder app that manages your forwarder outputs. You can set up multiple apps if you require different outputs.conf settings depending on the location in your network. Manage what forwarders can connect with whitelists and blacklists.

Same applies to inputs - Anything that is universal for all hosts of one type should be in one app. Anything that is specific to your one forwarder should then be moved to another. Again, manage with whitelists and blacklists.

The idea with your deployment server is to eventually get to a point where adding new systems, inputs and servers into splunk is as easy as pointing the new install to the deployment server. With this in mind, a little planning is required to make sure your serverclass.conf file is easy to read and logical. Your naming scheme for splunk instances becomes very important too!

I hope this helps.
This is by far no means a comprehensive guide to the deployment server.

[Brandon_ganem1]

The idea is you want to use apps on your deployserver. This requires some setup at the start, but you want to end up with no config in system/local and all of your forwarder config in apps deployed through your deployment server.

The reason for this is simple: re-usability. If you have multiple servers with the same inputs, you would put them in one app, and apply the configuration to all these servers through the deploy server. Then, in the case of a change, you make one change in one loc instead of on each forwarder.

So on your forwarder, all the cfg is in etc/apps/YOURAPPNAME

[krussell101]

It's interesting to me that if a forwarder is configured as a deployment client that splunk would not completely ignore what's on that host locally . . . When the deployment client gets its configuration from the deployment server where does it put it? What directory(ies)?

[Brandon_ganem1]

Its an order of precedence issue - the system/local config has the greatest precedence. You would need to replicate the configin the deploy server, then move the system/local config out of the way once you are sure everything is working.

I'd recommend doing a test on a system with test data. Install a forwarder and configure it as you normally would. Replicate the config to the deploy server, move the system/local out of the way and restart the forwarder. Make sure it works as expected.

I believe the terminology for "apps" that are not really apps would be Technology add-ons or TAs

[krussell101]

A follow on question . . .

If I have a forwarder configured as a deployment client but there is still a conf file in $HOME_SPLUNK/etc/system/local on that forwarder, will that file still be used? Or completely disregarded since it's supposed to be getting its configuration from the deployment server?

[krussell101]

Kool-aid.

I started the deployment last weekend and realized half way in that I had some open questions so I backed everything out.

This week has been planning. I'm considerably better versed now in "apps" (that aren't actually apps) and precedence.

But, still want to take baby steps. So I'm happy to hear that nothing can go awry by simply setting up my deployment server.

Thank you for your thorough response and suggestions. They'll go into my planning.