We are trying to switch to different deployment server from the current one. We have 100s of forwarders where deploymentclient.conf need to be changed.
Is it possible to do it via an app pushing to all the forwarders which will replace the deploymentclient.conf?
If the current deploymentclient.conf file is in $SPLUNK_HOME/etc/system/local then you cannot replace it with an app and are stuck using a manual/Ansible/Puppet solution to replace the file. Some brave souls have had luck writing a script in an app that deletes etc/system/local/deploymentclient.conf so the app instance of the file can take over.
If, however, Best Practices were followed and the current deploymentclient.conf file is in an app then you can modify the file and it will be pushed out as forwarders phone home.
Deploymentclient.conf is inside etc/system/local. I am one of the brave souls who attempted to remove deploymentclient.conf from system/local, but once you remove the system/local/deploymenclient.conf than a restart is required.
I tried restarting splunkforwarder after deleting the file, but splunk never comes back, it stops and stays in that state until I go and manually restart it.
When you delete etc/system/local/deploymentclient.conf you must create a deploymentclient.conf file in etc/apps. Without that, the forwarder has no way to contact the DS.
so the app contains
here inputs.conf triggers the script dsconfig.sh which deletes already existing deploymentclient.conf from etc/system/local and tries to restart splunkforwarder in order to new deploymenclient.conf(etc/apps/dsconfig/local/depoymentclient.conf) can take effect
But, restart from the script doesn't work. it only stops the forwarder and i have to manually login to fowarder to start it again and the new deploymentclient can take effect.
Do you know why splunkforwarder cannot be restarted using a script inside one of it's apps?
I'm not sure why the scripted restart doesn't work.
Have you tried telling the UF to restart itself when it installs the app? It's done using the "Restart splunkd" setting in the DS.
The best way to do this on many hosts is to use a server automation tool such as bladelogic or to do it directly via global shell or something similar.