Deployment Architecture

Splunk 6.5.1 : Search Head Cluster deployment changing default app.conf in user-prefs violating system-provided install manifest

New Member

I have been trying to clear an alert on a search head cluster that complains about :

File Integrity checks found 1 files that did not match the system-provided manifest. See splunkd.log for details.

Turns out the file is $SPLUNK_HOME/etc/apps/user-prefs/default/app.conf :

01-18-2017 14:42:00.136 +0800 WARN InstalledFilesHashChecker - An installed file="/opt/splunk/etc/apps/user-prefs/default/app.conf" did not pass hash-checking due to reason="content mismatch"

So I went and checked and set it to the standard 6.5.1 default file within the $SPLUNK_HOME/etc/shcluster/apps/user-prefs/default/app.conf on the search head deployment server. ( recently upgraded from 6.3.4 )

Once I run a SH cluster deploy splunk adds the following line to the $SPLUNK_HOME/etc/apps/user-prefs/default/app.conf that gets copied to each search head :

install_source_checksum = a9cff524a35e46b2e2a58a0a0129b3354066e789

Which is different to the mainifest in /opt/splunk/splunk-6.5.1-f74036626f0c-linux-2.6-x86_64-manifest :
f 444 splunk splunk splunk/etc/apps/user-prefs/default/app.conf ac9ff5d098283488c186e9f7b7464f0e269c332eef70db6f560b9392d6289878

Therefore it's appears to be checksum fault due to file being different from the install file.

Great 😞

Even if you remove the offending line from app.conf the error disappears however the SH deployer overwrites it and error returns.

Does anyone have a workaround and can someone confirm it as a bug ?

Tags (1)
0 Karma

Splunk Employee
Splunk Employee

user_prefs should not be deployed via deployer remove shcluster/apps/user-prefs, deploy to peers, return to each peer and reinstall the rpm/tar to restore the missing files

A few other things to check
1. review all contents of shcluster/apps ensure install_source_checksum is not present in default|local/apps.conf for any deployed apps if you have to clean up deploy to the cluster after cleanup actions.
2. Make sure the SHC members are not the client of a deployment server, if they are (deploymentclient.conf) remove this file and run a rolling restart. find and remove the deployment client artifacts left in opt/splunk/var

0 Karma
Get Updates on the Splunk Community!

New Learning Videos on Topics Most Requested by You! Plus This Month’s New Splunk ...

Splunk Lantern is a customer success center that provides advice from Splunk experts on valuable data ...

How I Instrumented a Rust Application Without Knowing Rust

As a technical writer, I often have to edit or create code snippets for Splunk's distributions of ...

Splunk Community Platform Survey

Hey Splunk Community, Starting today, the community platform may prompt you to participate in a survey. The ...