Safest way to upgrade a deployment server?

[Lucas_K]

What is the safest way to upgrade an deployment server which has a large number of clients?

I just tried an upgrade our deployment server from v5.0.4 to v6.0 and due to an issue with tenants.conf (an entirely different can of worms) resulted in all clients deleting all of their apps including their own deploymentclient.conf stopping them from phoning home again. I had to restore from a backup to get it working again, and login to each individual client and manually create a deploymentclient.conf to get them to dial in.

I've since fixed my tenants issue and want to try again I don't want a repeat of what happened before. Documentation says "disable the deployment server". Is referencing the command "./splunk disable deploy-server"?

What does this actually do? What will clients actually do? I don't have the luxury of shutting down every single splunk instance that calls home as there are indexers and remote forwarders involved.

For added complexity it is also our license server.

So ... what is the best way to upgrade without clients deciding to cut themselves off at the knees?

ps: tenants issue is that the feature is actually disabled and not depreciated (as the docs would have me believe) in the currently available x86_64 linux .tgz build of Splunk v6.0.

[Lucas_K]

well i've tested it and in our environment it did nothing at all. Communication STILL occured after deployment server was disabled, app updates still went out when they shouldn't have.

I think this is due to tenants. By default all deployment servers use a tenants its hidden and set to "default". From testing the disable deploy-server command, it seems to only disable that one "tenant=default" group and not the service as a whole (as it would be expected).

[sciurus · Accepted solution]

If you configure your environment in such a way that you can push out a new DS (ie, make a deployment-server app which has nothing but a deploymentclient.conf), then you could:

1. Setup a temporary DS with all the apps and serverclass.conf
2. Use the original DS to deploy a new deploymentclient.conf file, pointing clients to the temporary one
3. Upgrade the original DS
4. Use the temporary DS to deploy a new deploymentclient.conf file, pointing clients (or a subset of clients, to test) to the newly upgraded one.

That depends on having the setup in place, firewalls, etc, etc though.

[Lucas_K]

yes, that is a nice way to do it.

Unfortunately for me it seems as if the deployment server in v6.0 is broken if you have certain deployment server configuration ( http://answers.splunk.com/answers/114446/serverclassconf-using-the-same-app-name-under-different-cla... ). Ticket has been logged so any other upgrades to v6.0 has been stopped also.

[Lucas_K]

Just ran up a couple of test instances. 1 a deployment server the other a client.

Test #1.
performed a "splunk disable deploy-server".
updating an deployed app.
do a reload.
app updates appear in clients. 😞
So communication between the server and client continued. Perhaps the disable requires a restart inbetween. Lets try that.

Test #2.
performed a "splunk disable deploy-server"
update a deployed app.
restart deployment server splunk instance.
app updates appear in clients. 😞

So it would seem that clients can still connect to a deployment server that has had this command performed on it.

Does this command even do anything at all?

[Lucas_K]

ok more testing.

It would seem that using a tenants.conf bypasses the splunk disable deploy-server altogether. If your using a tenants.conf it will always been enabled regardless of what you do 😐

[lukejadamec]

That is a good question. What will happen to the clients when the deployment server is disabled. Somebody should definitely test that.
I'm new to Splunk, but I'll bet that it has no effect on the clients because the server and client are separate systems. I think the client will simply report in the log that there is no connectivity to the server, and the server will be shut down until it is re-enabled.
But, somebody should test it.

[Lucas_K]

yes. I'd already read that.

With a very large number of apps i'd have to individually update each of those apps and then do a reload. Each of those clients will then do a restart. As indexers are involved it will result in a larger outage than the actual upgrade.

Also in that thread no answer was given to what the actual splunk disable deploy-server actually does.

[somesoni2]

Have a look at this, may be helpful.

http://answers.splunk.com/answers/40027/temporarily-disable-deployment-server