How to switch to a different deployment server?

[saurabh009]

Hi,

We are trying to switch to different deployment server from the current one. We have 100s of forwarders where deploymentclient.conf need to be changed.
Is it possible to do it via an app pushing to all the forwarders which will replace the deploymentclient.conf?

[96nick]

I'm actually doing this right now, and I also sought help on how to do this in a safe way. I know this thread has been necro'd but in case someone else comes across it:

I'm going to assume the following:

• Your forwarders have deploymentclient.conf in $SPLUNK_HOME/etc/system/local
• You'd like to follow best practices and place it in an app instead.

The unfortunate thing about your environment right now is that ../etc/system/local has precedence over ../etc/apps, so pushing an app out from your current deployment server won't work. I'd recommend doing the following:

1. Create a manual app on ALL of your forwarders with an app called zzz_deploymentdefault. The name of the app doesn't matter as long as it starts with a 'z'. The app will contain a local directory with your deploymentclient.conf. This will not be from your OLD deployment server, but you will manually place it on the forwarders.
   1. In order to get it to your forwarders, create a script that copies the file into ../etc/apps and also renames/removes the /etc/system/local deploymentclient.conf. I'd highly recommend having a test server out there with a Splunk forwarder on it to make sure your script is right.
2. Now that all of your forwarders are connected to the NEW deployment server via that manual app you placed on the servers, create an app on your NEW deployment server that matches the contents of that manually pushed app. All of your forwarders will get this one too. I'd recommend naming the app with a capital letter (FritoLay_deploymentserver) as an app starting with a capital will take precedence over your zzz_deploymentdefault app.
3. Once you push that app out to all of your forwarders you are done!

How does this work?

It all has to do with precedence and having a safety net. The issue with only pushing an app from your deployment server is that if you make a change to your serverclass.conf that for whatever reason takes your deployment server-pushed app away, that server is gone! The manually placed app with deploymentclient.conf is your backup in case that ever happens.

There are downsides to this plan too. Changing your deployment server again will be more administratively challenging, and setting this up in the first place is more administratively difficult. You also would have to script your future forwarder deployments to include this zzz_deploymentdefault app.

[sonny_monti]

1. Open the firewall to the UFs on the new DS
2. On the UFs, change the $SPLUNK_HOME/etc/system/local/deploymentclient.conf to the following: [target-broker:deploymentServer] targetUri=(your new DS REST URI identifier) [deployment-client] clientName=(your UF client name)
3. Restart the UF
4. When you finish, launch splunk reload deploy-server on the DS

The best way to do this on many hosts is to use a server automation tool such as bladelogic or to do it directly via global shell or something similar.

[richgalloway]

If the current deploymentclient.conf file is in $SPLUNK_HOME/etc/system/local then you cannot replace it with an app and are stuck using a manual/Ansible/Puppet solution to replace the file. Some brave souls have had luck writing a script in an app that deletes etc/system/local/deploymentclient.conf so the app instance of the file can take over.

If, however, Best Practices were followed and the current deploymentclient.conf file is in an app then you can modify the file and it will be pushed out as forwarders phone home.

[jonwick]

@richgalloway 

Can I change the forwarder license as well???

We have bought new Splunk Enterprise License (not renewed) and hence I'm expecting we may have new forwarder license for UFs.

How this can be done?

[isoutamo]

UFs haven't any licenses so no need to change it. All licensing is handled on indexer levels (and/or HF level if you are storing also there).
r. Ismo

[jonwick]

Thanks @isoutamo ,

According to this https://docs.splunk.com/Documentation/Splunk/8.0.6/Admin/TypesofSplunklicenses

UF contains forwarder license, do we need to change this as well over existing agents.

We bought new Enterprise license because we are taking control over existing agents from other organisations (our Splunk Service Provider).

So if we keep older one this may cause UF owning issue. Installing and de installation stand ups to be unnecessary additional efforts.

[isoutamo]

Yes, you are right, UF contains that forwarder license, but it's actually just enable UF's basic features and it's not tight anyhow to indexing capacity or even Customer which are using it. So no need to do anything for it.
r. Ismo

[jonwick]

Thanks @isoutamo ,since it isn't paid I assume there are no validity to the license or any expiry threat.

Only things I need to perform is to change admin password and destination deployment server. Am I missing anything more?

And Just one last question, suppose we upgrade UF with our new UF later on, no trade off issues right??

[isoutamo]

I suppose that no expiration, but you should update it when needed. That usually extend licenses and certificates if those have validity periods.
One think is to clean fish buckets before you start to collect logs to the new target (if needed).

[jonwick]

Thanks a lot @isoutamo 

[saurabh009]

Deploymentclient.conf is inside etc/system/local. I am one of the brave souls who attempted to remove deploymentclient.conf from system/local, but once you remove the system/local/deploymenclient.conf than a restart is required.
I tried restarting splunkforwarder after deleting the file, but splunk never comes back, it stops and stays in that state until I go and manually restart it.

[richgalloway]

When you delete etc/system/local/deploymentclient.conf you must create a deploymentclient.conf file in etc/apps. Without that, the forwarder has no way to contact the DS.

[saurabh009]

so the app contains
etc/apps/dsconfig/default/inputs.conf
etc/apps/dsconfig/bin/dsconfig.sh
etc/apps/dsconfig/local/deploymentclient.conf

here inputs.conf triggers the script dsconfig.sh which deletes already existing deploymentclient.conf from etc/system/local and tries to restart splunkforwarder in order to new deploymenclient.conf(etc/apps/dsconfig/local/depoymentclient.conf) can take effect
But, restart from the script doesn't work. it only stops the forwarder and i have to manually login to fowarder to start it again and the new deploymentclient can take effect.

Do you know why splunkforwarder cannot be restarted using a script inside one of it's apps?

[richgalloway]

I'm not sure why the scripted restart doesn't work.
Have you tried telling the UF to restart itself when it installs the app? It's done using the "Restart splunkd" setting in the DS.

[ddrillic]

Where is your deploymentclient.conf on the forwarder? Is it under $SPLUNK_HOME/etc/system/local?