Deployment Server Uninstalling App from UF

[nieminej]

We have clustered Deployment Servers (with NFS shared drive) because we have total of clients tens of thousands at the final situation and we have deployed UF to Workstations and we have workstaion serverclass and few apps on it, including base_app which includes deploymentclient.conf, outputs.conf, server.conf and certificates.

And when UF Agent is installed to Worstations trought SCCM it phoneshome and then it just tells Serverclass=workstations is uninstalling app=C:\ProgramFiles\SplunkUniversalForwarder\etc\apps\base_uf

There is crossServerChecksum tried with true or false and no changes. We can't figure out it from any logs or so, there is nothing errors it just tells that it started to uninstall app and then restarts UF and loses connections.

If we check one unique client it belongs only to one Serverclass, and Worstations Serverclass include our base_app and then Splunk_TA_windows and sysmon apps.

We have version 9.4.1 on our Enterprise and UF's have 9.3.2, phonehomes coming trough F5 LoadBalancer.

We are running out of ideas with this. 

[nieminej]

Well we ended up to break down the deployment cluster and use dedicated DS and divide clients to then because we have a deadline which need to reach. 

We figured out that problem might be in NFS shared drive which issued that DS1 had only working hashes and everytime client phoned to DS2 it lost apps because mismatch on checksums and it had no reference to any apps for that specific client and then just uninstalled it. Don't know for sure but its not a significant problem anymore because we changed the architecture.

[livehybrid]

Hi @nieminej 

When the UFs are installed, do they come from an image with the UF installed and you initialise it somehow, or is it a vanilla install?

It sounds like you are using SCCM to install an app which the DS think it controls? If it was me, I'd have a bare-bones Deploymentclient app with low precedence (e.g. z_myorg_deployclient) which has your deploymentclient.conf - deploy this using SCCM and then when it connects to the DS it should pull down the base_uf app - this has a higher precedence that z_myorg_deployclient so the deploymentclient.conf here will take over, allowing you to make updates in the future if needed. 

I would definitely avoid having an app controlled by DS *and* pre-installed on the UF.

Please let me know how you get on and consider adding karma to this or any other answer if it has helped.
Regards

Will 

[kiran_panchavat]

@nieminej 

Confirm that the app is consistently named base_app (not base_uf) across the deployment server, UF directory (C:\Program Files\SplunkUniversalForwarder\etc\apps), and serverclass.conf. 

Check the contents of base_app on the deployment server ($SPLUNK_HOME/etc/deployment-apps/base_app) and ensure it matches what’s expected on the UF after deployment.

Manually install the UF on a test workstation (bypassing SCCM) and configure it to phone home to the deployment server. Does the issue persist? This isolates whether SCCM is a factor.

I suspect the deployment server is instructing the UF to uninstall base_app due to either, A mismatch between the app’s expected state (as defined in serverclass.conf) and its actual state on the UF after SCCM deployment (OR) A misconfiguration in base_app’s configs causing the UF to misinterpret its deployment instructions.