I have been trying to clear an alert on a search head cluster that complains about :
File Integrity checks found 1 files that did not match the system-provided manifest. See splunkd.log for details.
Turns out the file is $SPLUNK_HOME/etc/apps/user-prefs/default/app.conf :
01-18-2017 14:42:00.136 +0800 WARN InstalledFilesHashChecker - An installed file="/opt/splunk/etc/apps/user-prefs/default/app.conf" did not pass hash-checking due to reason="content mismatch"
So I went and checked and set it to the standard 6.5.1 default file within the $SPLUNK_HOME/etc/shcluster/apps/user-prefs/default/app.conf on the search head deployment server. ( recently upgraded from 6.3.4 )
Once I run a SH cluster deploy splunk adds the following line to the $SPLUNK_HOME/etc/apps/user-prefs/default/app.conf that gets copied to each search head :
Which is different to the mainifest in /opt/splunk/splunk-6.5.1-f74036626f0c-linux-2.6-x86_64-manifest : f 444 splunk splunk splunk/etc/apps/user-prefs/default/app.conf ac9ff5d098283488c186e9f7b7464f0e269c332eef70db6f560b9392d6289878
Therefore it's appears to be checksum fault due to file being different from the install file.
Even if you remove the offending line from app.conf the error disappears however the SH deployer overwrites it and error returns.
Does anyone have a workaround and can someone confirm it as a bug ?
user_prefs should not be deployed via deployer remove shcluster/apps/user-prefs, deploy to peers, return to each peer and reinstall the rpm/tar to restore the missing files
A few other things to check
1. review all contents of shcluster/apps ensure install_source_checksum is not present in default|local/apps.conf for any deployed apps if you have to clean up deploy to the cluster after cleanup actions.
2. Make sure the SHC members are not the client of a deployment server, if they are (deploymentclient.conf) remove this file and run a rolling restart. find and remove the deployment client artifacts left in opt/splunk/var