i have seen this happen before where the deploymentclient.conf is initially place under /etc/apps/local/deploymentclient.conf
and then once the client has established connection with deployment server (phones home), the same app is configured on the deployment server and pushed out to client which overrides the initial app () and this is what causes the breaking as splunk warns against this in below documentation snippet:
Important: Do not use the deployment server to push deploymentclient.conf updates to the deployment clients. Doing so can potentially lead to situations where the deployment clients lose their ability to contact the deployment server.
You should also check for this kind of error as splunk tries to stack the app containing the deploymentclient.conf configs on top of each other due to the app being wrongly pushed out via deployment sever as warned against above:
ERROR DeployedApplication - untarToDest, failed to mkdir opt/splunkforwarder/etc/apps/<my_app>/<my_app>/<my_app>/<my_app>/<my_app>/local: Permission denied
To fix this issue, What we did we removed the app with deploymentclient.conf configs from the deployment server and re-configured the deployment client app again on the remote server or ran
./splunk set deploy-poll <IP_address/hostname>:<management_port>