Deployment Architecture

Safest way to upgrade a deployment server?

Lucas_K
Motivator

What is the safest way to upgrade an deployment server which has a large number of clients?

I just tried an upgrade our deployment server from v5.0.4 to v6.0 and due to an issue with tenants.conf (an entirely different can of worms) resulted in all clients deleting all of their apps including their own deploymentclient.conf stopping them from phoning home again. I had to restore from a backup to get it working again, and login to each individual client and manually create a deploymentclient.conf to get them to dial in.

I've since fixed my tenants issue and want to try again I don't want a repeat of what happened before. Documentation says "disable the deployment server". Is referencing the command "./splunk disable deploy-server"?

What does this actually do? What will clients actually do? I don't have the luxury of shutting down every single splunk instance that calls home as there are indexers and remote forwarders involved.

For added complexity it is also our license server.

So ... what is the best way to upgrade without clients deciding to cut themselves off at the knees?

ps: tenants issue is that the feature is actually disabled and not depreciated (as the docs would have me believe) in the currently available x86_64 linux .tgz build of Splunk v6.0.

1 Solution

sciurus
Path Finder

If you configure your environment in such a way that you can push out a new DS (ie, make a deployment-server app which has nothing but a deploymentclient.conf), then you could:

  1. Setup a temporary DS with all the apps and serverclass.conf
  2. Use the original DS to deploy a new deploymentclient.conf file, pointing clients to the temporary one
  3. Upgrade the original DS
  4. Use the temporary DS to deploy a new deploymentclient.conf file, pointing clients (or a subset of clients, to test) to the newly upgraded one.

That depends on having the setup in place, firewalls, etc, etc though.

View solution in original post

Lucas_K
Motivator

well i've tested it and in our environment it did nothing at all. Communication STILL occured after deployment server was disabled, app updates still went out when they shouldn't have.

I think this is due to tenants. By default all deployment servers use a tenants its hidden and set to "default". From testing the disable deploy-server command, it seems to only disable that one "tenant=default" group and not the service as a whole (as it would be expected).

sciurus
Path Finder

If you configure your environment in such a way that you can push out a new DS (ie, make a deployment-server app which has nothing but a deploymentclient.conf), then you could:

  1. Setup a temporary DS with all the apps and serverclass.conf
  2. Use the original DS to deploy a new deploymentclient.conf file, pointing clients to the temporary one
  3. Upgrade the original DS
  4. Use the temporary DS to deploy a new deploymentclient.conf file, pointing clients (or a subset of clients, to test) to the newly upgraded one.

That depends on having the setup in place, firewalls, etc, etc though.

Lucas_K
Motivator

yes, that is a nice way to do it.

Unfortunately for me it seems as if the deployment server in v6.0 is broken if you have certain deployment server configuration ( http://answers.splunk.com/answers/114446/serverclassconf-using-the-same-app-name-under-different-cla... ). Ticket has been logged so any other upgrades to v6.0 has been stopped also.

Lucas_K
Motivator

Just ran up a couple of test instances. 1 a deployment server the other a client.

Test #1.
performed a "splunk disable deploy-server".
updating an deployed app.
do a reload.
app updates appear in clients. 😞
So communication between the server and client continued. Perhaps the disable requires a restart inbetween. Lets try that.

Test #2.
performed a "splunk disable deploy-server"
update a deployed app.
restart deployment server splunk instance.
app updates appear in clients. 😞

So it would seem that clients can still connect to a deployment server that has had this command performed on it.

Does this command even do anything at all?

0 Karma

Lucas_K
Motivator

ok more testing.

It would seem that using a tenants.conf bypasses the splunk disable deploy-server altogether. If your using a tenants.conf it will always been enabled regardless of what you do 😐

0 Karma

lukejadamec
Super Champion

That is a good question. What will happen to the clients when the deployment server is disabled. Somebody should definitely test that.
I'm new to Splunk, but I'll bet that it has no effect on the clients because the server and client are separate systems. I think the client will simply report in the log that there is no connectivity to the server, and the server will be shut down until it is re-enabled.
But, somebody should test it.

0 Karma

Lucas_K
Motivator

yes. I'd already read that.

With a very large number of apps i'd have to individually update each of those apps and then do a reload. Each of those clients will then do a restart. As indexers are involved it will result in a larger outage than the actual upgrade.

Also in that thread no answer was given to what the actual splunk disable deploy-server actually does.

0 Karma

somesoni2
Revered Legend
0 Karma
Get Updates on the Splunk Community!

Take Your Breath Away with Splunk Risk-Based Alerting (RBA)

WATCH NOW!The Splunk Guide to Risk-Based Alerting is here to empower your SOC like never before. Join Haylee ...

Industry Solutions for Supply Chain and OT, Amazon Use Cases, Plus More New Articles ...

Splunk Lantern is a Splunk customer success center that provides advice from Splunk experts on valuable data ...

Enterprise Security Content Update (ESCU) | New Releases

In November, the Splunk Threat Research Team had one release of new security content via the Enterprise ...