Using the Deployer to deploy Apps to my Search Heads in a SHC setup. I had been trying to push out a new App, Phantom to all the SH but strangely, the older version of the App kept being installed.
What was done?
On Deployer, removed Phantom App and push out bundle. On all 3 SHs, the App did get removed correctly.
Unzip new Phantom and push out bundle. On all 3 SHs, the Phantom App appears but the App was a previous version.
e.g. On the new Phantom App, I have a "local" folder in it and there is no such folder in the previous version of Phantom App. However, even though the bundle push was successful, the App deployed has no "local" folder and have old configurations as well. This means that the new Phantom was actually not in place.
Tried other Apps and observed similar issue
For app directories only, all files placed under both default and local subdirectories get merged into default subdirectories on the members, post-deployment(see attachment for more details).The deployer never deploys files to the members' local app directories, $SPLUNK_HOME/etc/apps//local. Instead, it deploys both local and default settings from the configuration bundle to the members' default app directories, $SPLUNK_HOME/etc/apps//default. This ensures that deployed settings never overwrite local or replicated runtime settings on the members. Otherwise, for example, app upgrades would wipe out runtime changes.