Why is the Splunk universal forwarder dropping from deployment server?

[rameshtdp]

Splunk UF's are having different versions 6.0.0, 6.3 and 6.5.2 are connecting to Deployment server with 7.2.6 server.
All of sudden some of the clients are dropping from Deployment server.

[mguhad · Accepted solution]

Hi,
i have seen this happen before where the deploymentclient.conf is initially place under /etc/apps/local/deploymentclient.conf
and then once the client has established connection with deployment server (phones home), the same app is configured on the deployment server and pushed out to client which overrides the initial app () and this is what causes the breaking as splunk warns against this in below documentation snippet:
https://docs.splunk.com/Documentation/Splunk/7.2.6/Updating/Configuredeploymentclients

Important: Do not use the deployment server to push deploymentclient.conf updates to the deployment clients. Doing so can potentially lead to situations where the deployment clients lose their ability to contact the deployment server.

You should also check for this kind of error as splunk tries to stack the app containing the deploymentclient.conf configs on top of each other due to the app being wrongly pushed out via deployment sever as warned against above:

ERROR DeployedApplication - untarToDest, failed to mkdir opt/splunkforwarder/etc/apps/<my_app>/<my_app>/<my_app>/<my_app>/<my_app>/local: Permission denied

To fix this issue, What we did we removed the app with deploymentclient.conf configs from the deployment server and re-configured the deployment client app again on the remote server or ran ./splunk set deploy-poll <IP_address/hostname>:<management_port>

[rameshtdp]

Thanks mguhad!