topic Re: How to switch to a different deployment server? in Deployment Architecture

How to switch to a different deployment server?

saurabh009 — Wed, 29 May 2019 15:06:10 GMT

Hi,

We are trying to switch to different deployment server from the current one. We have 100s of forwarders where deploymentclient.conf need to be changed.
Is it possible to do it via an app pushing to all the forwarders which will replace the deploymentclient.conf?

Re: How to switch to a different deployment server?

ddrillic — Wed, 29 May 2019 15:17:28 GMT

Where is your deploymentclient.conf on the forwarder? Is it under $SPLUNK_HOME/etc/system/local?

Re: How to switch to a different deployment server?

richgalloway — Wed, 29 May 2019 15:20:52 GMT

If the current deploymentclient.conf file is in $SPLUNK_HOME/etc/system/local then you cannot replace it with an app and are stuck using a manual/Ansible/Puppet solution to replace the file. Some brave souls have had luck writing a script in an app that deletes etc/system/local/deploymentclient.conf so the app instance of the file can take over.

If, however, Best Practices were followed and the current deploymentclient.conf file is in an app then you can modify the file and it will be pushed out as forwarders phone home.

Re: How to switch to a different deployment server?

saurabh009 — Wed, 29 May 2019 16:05:13 GMT

Deploymentclient.conf is inside etc/system/local. I am one of the brave souls who attempted to remove deploymentclient.conf from system/local, but once you remove the system/local/deploymenclient.conf than a restart is required.
I tried restarting splunkforwarder after deleting the file, but splunk never comes back, it stops and stays in that state until I go and manually restart it.

Re: How to switch to a different deployment server?

richgalloway — Wed, 29 May 2019 19:11:02 GMT

When you delete etc/system/local/deploymentclient.conf you must create a deploymentclient.conf file in etc/apps. Without that, the forwarder has no way to contact the DS.

Re: How to switch to a different deployment server?

saurabh009 — Thu, 30 May 2019 15:38:48 GMT

so the app contains
etc/apps/dsconfig/default/inputs.conf
etc/apps/dsconfig/bin/dsconfig.sh
etc/apps/dsconfig/local/deploymentclient.conf

here inputs.conf triggers the script dsconfig.sh which deletes already existing deploymentclient.conf from etc/system/local and tries to restart splunkforwarder in order to new deploymenclient.conf(etc/apps/dsconfig/local/depoymentclient.conf) can take effect
But, restart from the script doesn't work. it only stops the forwarder and i have to manually login to fowarder to start it again and the new deploymentclient can take effect.

Do you know why splunkforwarder cannot be restarted using a script inside one of it's apps?

Re: How to switch to a different deployment server?

richgalloway — Fri, 31 May 2019 12:44:23 GMT

I'm not sure why the scripted restart doesn't work.
Have you tried telling the UF to restart itself when it installs the app? It's done using the "Restart splunkd" setting in the DS.

Re: How to switch to a different deployment server?

sonny_monti — Fri, 31 May 2019 14:22:16 GMT

Open the firewall to the UFs on the new DS
On the UFs, change the $SPLUNK_HOME/etc/system/local/deploymentclient.conf to the following: [target-broker:deploymentServer] targetUri=(your new DS REST URI identifier) [deployment-client] clientName=(your UF client name)
Restart the UF
When you finish, launch splunk reload deploy-server on the DS

The best way to do this on many hosts is to use a server automation tool such as bladelogic or to do it directly via global shell or something similar.

Re: How to switch to a different deployment server?

jonwick — Wed, 30 Sep 2020 16:57:51 GMT

@richgalloway

Can I change the forwarder license as well???

We have bought new Splunk Enterprise License (not renewed) and hence I'm expecting we may have new forwarder license for UFs.

How this can be done?

Re: How to switch to a different deployment server?

isoutamo — Wed, 30 Sep 2020 17:01:35 GMT

UFs haven't any licenses so no need to change it. All licensing is handled on indexer levels (and/or HF level if you are storing also there).
r. Ismo

Re: How to switch to a different deployment server?

jonwick — Wed, 30 Sep 2020 17:08:29 GMT

Thanks @isoutamo ,

According to this https://docs.splunk.com/Documentation/Splunk/8.0.6/Admin/TypesofSplunklicenses

UF contains forwarder license, do we need to change this as well over existing agents.

We bought new Enterprise license because we are taking control over existing agents from other organisations (our Splunk Service Provider).

So if we keep older one this may cause UF owning issue. Installing and de installation stand ups to be unnecessary additional efforts.

Re: How to switch to a different deployment server?

isoutamo — Wed, 30 Sep 2020 17:11:28 GMT

Yes, you are right, UF contains that forwarder license, but it's actually just enable UF's basic features and it's not tight anyhow to indexing capacity or even Customer which are using it. So no need to do anything for it.
r. Ismo

Re: How to switch to a different deployment server?

jonwick — Wed, 30 Sep 2020 17:46:51 GMT

Thanks @isoutamo ,since it isn't paid I assume there are no validity to the license or any expiry threat.

Only things I need to perform is to change admin password and destination deployment server. Am I missing anything more?

And Just one last question, suppose we upgrade UF with our new UF later on, no trade off issues right??

Re: How to switch to a different deployment server?

96nick — Wed, 30 Sep 2020 17:59:36 GMT

I'm actually doing this right now, and I also sought help on how to do this in a safe way. I know this thread has been necro'd but in case someone else comes across it:

I'm going to assume the following:

Your forwarders have deploymentclient.conf in $SPLUNK_HOME/etc/system/local
You'd like to follow best practices and place it in an app instead.

The unfortunate thing about your environment right now is that ../etc/system/local has precedence over ../etc/apps, so pushing an app out from your current deployment server won't work. I'd recommend doing the following:

Create a manual app on ALL of your forwarders with an app called zzz_deploymentdefault. The name of the app doesn't matter as long as it starts with a 'z'. The app will contain a local directory with your deploymentclient.conf. This will not be from your OLD deployment server, but you will manually place it on the forwarders.
1. In order to get it to your forwarders, create a script that copies the file into ../etc/apps and also renames/removes the /etc/system/local deploymentclient.conf. I'd highly recommend having a test server out there with a Splunk forwarder on it to make sure your script is right.
Now that all of your forwarders are connected to the NEW deployment server via that manual app you placed on the servers, create an app on your NEW deployment server that matches the contents of that manually pushed app. All of your forwarders will get this one too. I'd recommend naming the app with a capital letter (FritoLay_deploymentserver) as an app starting with a capital will take precedence over your zzz_deploymentdefault app.
Once you push that app out to all of your forwarders you are done!

How does this work?

It all has to do with precedence and having a safety net. The issue with only pushing an app from your deployment server is that if you make a change to your serverclass.conf that for whatever reason takes your deployment server-pushed app away, that server is gone! The manually placed app with deploymentclient.conf is your backup in case that ever happens.

There are downsides to this plan too. Changing your deployment server again will be more administratively challenging, and setting this up in the first place is more administratively difficult. You also would have to script your future forwarder deployments to include this zzz_deploymentdefault app.

Re: How to switch to a different deployment server?

isoutamo — Wed, 30 Sep 2020 18:16:55 GMT

I suppose that no expiration, but you should update it when needed. That usually extend licenses and certificates if those have validity periods.
One think is to clean fish buckets before you start to collect logs to the new target (if needed).

Re: How to switch to a different deployment server?

jonwick — Wed, 30 Sep 2020 18:30:24 GMT

Thanks a lot @isoutamo