We have a Splunk cluster deployment with this settings:
1 Node: Master and Deployment Server
2 Nodes: Indexers
1 Node: Search Head
Using default ports 8089 for management and 9997 for data forwarding.
As our enviroment is getting larger and in the future we are going to hit the 500 UF managed by the Deployment Server, we are thinking in moving the Deployment Server to a separate instance in the same server.
The only issue I see here is the need of use a different port for the new instance (8090 for example) and configure all the existing UF to talk to the new Deployment Server by the new port.
Is there any other possible issues I'm missing?
The different port number and the actual changeover you mention are the two largest issues. Actually moving the deployment server (technically making a new instance that is a deployment server) is not terribly difficult. You will need to do some work to make sure your multiple deployment servers have the same apps and so on - a git / hg / subversion repository would work well here.
There are a couple of pieces of advice from the community on this at http://wiki.splunk.com/Things_I_wish_I_knew_then .
First of all, don't configure your deployment server connection in
etc/system/local/deploymentclient.conf. Use an app that tells your systems how to find their deployment server. You can then push updates to that app from Deployment Server A to shuffle off a set of clients to Deployment Server B.
Second, use DNS CNAME(s) for your deployment server(s) instead of specific server names so you can move it easily later if you need to.
A somewhat reasonable approach - if your box is powerful enough - is to run 2 (or more) instances of deployment server on the same node. Make them all listen on the same PORT, but a different IP using
We had an app that had a deploymentclient.conf for the above like scenario. We ran into a problem and built another deployment server to replace the existing one with same IP etc.
Upon rebooting the VM with the new Splunk instance installed and the old instance disabled, we had not yet copied the deployment apps, nor the serverclass.conf.
So all of the deployment clients connected AND PROCEEDED TO DELETE ALL OF THEIR APPS AND HENCE NO MORE DEPLOYMENT.CLIENT OR ANY MORE PHONE HOMES!!!
This is what happens when you mange your deploymentclient.conf in an app and accidently start a new unconfigured, deployment server instance in place.
We had to go and redo 425 SUFs and almost closed our business.
DO NOT DO THIS!
Splunk braindead deployment client behaviour is horrible. They should check if the deployment server is configured and not assume because no apps were found to delete EVERYTHING!!!