Splunk 6.5.1 : Search Head Cluster deployment changing default app.conf in user-prefs violating system-provided install manifest

[ranton5000]

I have been trying to clear an alert on a search head cluster that complains about :

File Integrity checks found 1 files that did not match the system-provided manifest. See splunkd.log for details.

Turns out the file is $SPLUNK_HOME/etc/apps/user-prefs/default/app.conf :

01-18-2017 14:42:00.136 +0800 WARN InstalledFilesHashChecker - An installed file="/opt/splunk/etc/apps/user-prefs/default/app.conf" did not pass hash-checking due to reason="content mismatch"

So I went and checked and set it to the standard 6.5.1 default file within the $SPLUNK_HOME/etc/shcluster/apps/user-prefs/default/app.conf on the search head deployment server. ( recently upgraded from 6.3.4 )

Once I run a SH cluster deploy splunk adds the following line to the $SPLUNK_HOME/etc/apps/user-prefs/default/app.conf that gets copied to each search head :

install_source_checksum = a9cff524a35e46b2e2a58a0a0129b3354066e789

Which is different to the mainifest in /opt/splunk/splunk-6.5.1-f74036626f0c-linux-2.6-x86_64-manifest :
f 444 splunk splunk splunk/etc/apps/user-prefs/default/app.conf ac9ff5d098283488c186e9f7b7464f0e269c332eef70db6f560b9392d6289878

Therefore it's appears to be checksum fault due to file being different from the install file.

Great 😞

Even if you remove the offending line from app.conf the error disappears however the SH deployer overwrites it and error returns.

Does anyone have a workaround and can someone confirm it as a bug ?

[rfaircloth_splu]

user_prefs should not be deployed via deployer remove shcluster/apps/user-prefs, deploy to peers, return to each peer and reinstall the rpm/tar to restore the missing files

A few other things to check
1. review all contents of shcluster/apps ensure install_source_checksum is not present in default|local/apps.conf for any deployed apps if you have to clean up deploy to the cluster after cleanup actions.
2. Make sure the SHC members are not the client of a deployment server, if they are (deploymentclient.conf) remove this file and run a rolling restart. find and remove the deployment client artifacts left in opt/splunk/var