File Integrity checks found files that did not match the system-provided manifest?

[Gregski11]

Splunk version 9.0.0 on Windows servers

Please allow me to preface this by saying yes I GOOGLED this error and yes I did find some hits on this very Community site though 6 years old, and yes I did follow the many links to other links, which in turn lead me to more orphaned threads, ha ha, hence deciding to make this fresh more current post in which I hope we can present a solution

File Integrity checks found 40 files that did not match the system-provided manifest. Review the list of problems reported by the InstalledFileHashChecker in splunkd.log File Integrity Check View ; potentially restore files from installation media, change practices to avoid changing files, or work with support to identify the problem.

So if you click that link you get something like this:

List of installed files presenting integrity check failures

The table below shows files that were installed by the Splunk Enterprise package and have been improperly modified or are missing. Learn more.

 

Search is completed

 

File path Check result

List of installed files presenting integrity check failures
The table below shows files that were installed by the Splunk Enterprise package and have been improperly modified or are missing. Learn more.

Search is completed
File path Check result
C:\Program Files\Splunk\etc/apps/SplunkForwarder/default/app.conf missing
C:\Program Files\Splunk\etc/apps/SplunkForwarder/default/default-mode.conf missing
C:\Program Files\Splunk\etc/apps/SplunkForwarder/default/health.conf missing
C:\Program Files\Splunk\etc/apps/SplunkForwarder/default/outputs.conf missing
C:\Program Files\Splunk\etc/apps/SplunkForwarder/default/server.conf missing
C:\Program Files\Splunk\etc/apps/SplunkForwarder/metadata/default.meta missing
C:\Program Files\Splunk\etc/apps/python_upgrade_readiness_app/appserver/static/pages/jquery_scan.js differs
C:\Program Files\Splunk\etc/apps/python_upgrade_readiness_app/appserver/static/pages/python_scan.js differs
C:\Program Files\Splunk\etc/apps/python_upgrade_readiness_app/appserver/static/pages/setting.js differs
C:\Program Files\Splunk\etc/apps/python_upgrade_readiness_app/appserver/static/pages/setting_scan.js differs
C:\Program Files\Splunk\etc/apps/python_upgrade_readiness_app/appserver/static/pages/splunk9x_scan.js differs
C:\Program Files\Splunk\etc/apps/python_upgrade_readiness_app/bin/eura_app_list.py differs
C:\Program Files\Splunk\etc/apps/python_upgrade_readiness_app/bin/eura_check_mongodb_tls_dns_validation.py differs
C:\Program Files\Splunk\etc/apps/python_upgrade_readiness_app/bin/eura_check_search_peer_ssl_config.py differs
C:\Program Files\Splunk\etc/apps/python_upgrade_readiness_app/bin/eura_email_notification_switch_scripted_input.py differs
C:\Program Files\Splunk\etc/apps/python_upgrade_readiness_app/bin/eura_remote_latest_report.py differs
C:\Program Files\Splunk\etc/apps/python_upgrade_readiness_app/bin/eura_remote_scan_scripted_input.py differs
C:\Program Files\Splunk\etc/apps/python_upgrade_readiness_app/bin/eura_scan_apps.py differs
C:\Program Files\Splunk\etc/apps/python_upgrade_readiness_app/bin/eura_scan_deployment.py differs
C:\Program Files\Splunk\etc/apps/python_upgrade_readiness_app/bin/eura_scan_process.py differs
C:\Program Files\Splunk\etc/apps/python_upgrade_readiness_app/bin/eura_send_email.py differs
C:\Program Files\Splunk\etc/apps/python_upgrade_readiness_app/bin/eura_telemetry.py differs
C:\Program Files\Splunk\etc/apps/python_upgrade_readiness_app/bin/jura_scan_process.py differs
C:\Program Files\Splunk\etc/apps/python_upgrade_readiness_app/bin/jura_telemetry.py differs
C:\Program Files\Splunk\etc/apps/python_upgrade_readiness_app/bin/libs_py2/pura_libs_utils/pura_consts.py differs
C:\Program Files\Splunk\etc/apps/python_upgrade_readiness_app/bin/libs_py2/pura_libs_utils/pura_logger_manager.py differs
C:\Program Files\Splunk\etc/apps/python_upgrade_readiness_app/bin/libs_py2/pura_libs_utils/pura_skynet_log_manager.py differs
C:\Program Files\Splunk\etc/apps/python_upgrade_readiness_app/bin/libs_py2/pura_libs_utils/pura_utils.py differs
C:\Program Files\Splunk\etc/apps/python_upgrade_readiness_app/bin/libs_py2/pura_libs_utils/splunkbaseapps.csv differs
C:\Program Files\Splunk\etc/apps/python_upgrade_readiness_app/bin/libs_py3/pura_libs_utils/pura_consts.py differs
C:\Program Files\Splunk\etc/apps/python_upgrade_readiness_app/bin/libs_py3/pura_libs_utils/pura_logger_manager.py differs
C:\Program Files\Splunk\etc/apps/python_upgrade_readiness_app/bin/libs_py3/pura_libs_utils/pura_skynet_log_manager.py differs
C:\Program Files\Splunk\etc/apps/python_upgrade_readiness_app/bin/libs_py3/pura_libs_utils/pura_utils.py differs
C:\Program Files\Splunk\etc/apps/python_upgrade_readiness_app/bin/libs_py3/pura_libs_utils/splunkbaseapps.csv differs
C:\Program Files\Splunk\etc/apps/python_upgrade_readiness_app/bin/pura_remote_latest_report.py differs
C:\Program Files\Splunk\etc/apps/python_upgrade_readiness_app/bin/pura_telemetry.py differs
C:\Program Files\Splunk\etc/apps/python_upgrade_readiness_app/bin/scan_process.py differs
C:\Program Files\Splunk\etc/apps/python_upgrade_readiness_app/default/data/ui/nav/default.env_cloud.xml missing
C:\Program Files\Splunk\etc/apps/python_upgrade_readiness_app/default/data/ui/nav/default.xml differs
C:\Program Files\Splunk\etc/system/local/README missing

[spodda01da]

Had similar issue when I upgraded the app from version 1.0.0 to 4.0.2. 

Reverted back to version 1.0.0 which seems to fix the issue.

[Gregski11]

well this is rather peculiar after uninstalling the SplunkForwarder app properly through the Web UI, Splunk still looks for it and the missing subdirectory and all of it's contents still fail the File Integrity Check, this is so Bogus!

C:\Program Files\Splunk\etc/apps/SplunkForwarder/default/app.conf missing
C:\Program Files\Splunk\etc/apps/SplunkForwarder/default/default-mode.conf missing
C:\Program Files\Splunk\etc/apps/SplunkForwarder/default/health.conf missing
C:\Program Files\Splunk\etc/apps/SplunkForwarder/default/outputs.conf missing
C:\Program Files\Splunk\etc/apps/SplunkForwarder/default/server.conf missing
C:\Program Files\Splunk\etc/apps/SplunkForwarder/metadata/default.meta missing

So does this mean you can't delete this app on your Indexers for example? where the buck stops here 

[Gregski11]

forgive me for the lengthy post, the intent is to help somebody out in the future, so after manually replacing more files from a sister server we have progress

File Integrity checks found 26 files that did not match the system-provided manifest. 

so we are down from 40 down to only 26

here's the batch of 10 more I just replaced

bin/libs_py2/pura_libs_utils/pura_consts.py
bin/libs_py2/pura_libs_utils/pura_logger_manager.py
bin/libs_py2/pura_libs_utils/pura_skynet_log_manager.py
bin/libs_py2/pura_libs_utils/pura_utils.py
bin/libs_py2/pura_libs_utils/splunkbaseapps.csv

bin/libs_py3/pura_libs_utils/pura_consts.py
bin/libs_py3/pura_libs_utils/pura_logger_manager.py
bin/libs_py3/pura_libs_utils/pura_skynet_log_manager.py
bin/libs_py3/pura_libs_utils/pura_utils.py
bin/libs_py3/pura_libs_utils/splunkbaseapps.csv

yes it's a pain, yes I had to copy them over manually and restart Splunk 

[Gregski11]

well this is interesting and I think worth sharing

so I decided to move the python_upgrade_readiness_app to a new folder I created called Splunk\etc\apps we deleted in order to simulate deleting the app without actually deleting the app, made sense to me at the time

then I restarted Splunk, and whoa, it came back now saying we have 1,100 files that failed the integrity check, what on earth? talk about one step forward and a 1,000 steps back 

[PickleRick]

This simply means that some of the files that you are not supposed to touch were touched. That's it. What's the practical significance of this message? Depends on the files.

Just don't touch the things that come with splunk by default. If you want to disable app - disable it but don't just delete a whole folder.

[Gregski11]

Rick I agree, the question is how do we fix it after someone else has touched them, it wasn't me it was the prior admin, this may be the scenario for other admins on here as well who inherit the environment 

[PickleRick]

The easiest way would be probably to get an installer, unpack it somewhere and copy over the missing files (you might need to fix ownership as well depending on what user your splunk is installed/runs under)

[isoutamo]

Unless someone have changed configurations on default folder the easiest way to fix this issue to just reinstall the same version again. Then it just add those missing files and also replace changed files on default directories. 

Anyhow in your situation I just take backup of SPLUNK_HOME/etc before do reinstallation, just for case that there has change something else. 

r. Ismo

[Gregski11]

thank you all for trying to help and offering solutions, my manual fork lift replacement of those 5 files mentioned above did actually work (I just restarted Splunk on the wrong server at first, dope) 

my question now is why do I even need that app, can I just uninstall it and be done with it, and if and when we decide to upgrade Python we can just worry about reinstalling it then? 

[Gregski11]

alright so next I decided to divide and conquer, as you can see a lot of the headache is attributed to the python_upgrade_readiness_app which I would deem non life threatening so I decided to fandangle with it a little bit starting with these 5 "static pages" files 

C:\Program Files\Splunk\etc/apps/python_upgrade_readiness_app/appserver/static/pages/jquery_scan.js
C:\Program Files\Splunk\etc/apps/python_upgrade_readiness_app/appserver/static/pages/python_scan.js
C:\Program Files\Splunk\etc/apps/python_upgrade_readiness_app/appserver/static/pages/setting.js
C:\Program Files\Splunk\etc/apps/python_upgrade_readiness_app/appserver/static/pages/setting_scan.js
C:\Program Files\Splunk\etc/apps/python_upgrade_readiness_app/appserver/static/pages/splunk9x_scan.js

I simply renamed them all to origina_file_name.js.differsand copied the identical files from a sister server with no intergrity check issues and restarted Splunk 

[Gregski11]

so let's go after the low hanging fruit first:

C:\Program Files\Splunk\etc/system/local/README missing

the above README file is one that I myself indeed deleted, I simply did not want or need a REAME file cluttering up our /system/local subdirectories

I find it interesting that Splunk finds that a missing README file undermines the integrity of the system, it makes me question the validity of this alert in general, what are your thoughts?

[Gregski11]

so it appears that clicking that link pretty much does the same thing as running this command

splunk validate files

[Dallastek1]

Ive got a little over 4k that show missing for S:\Program Files\Splunk\share/splunk/search_mrsparkle/xxx
on my heavy forwarder. Im still trying to determine what mrsparkle does, I may unpack a new splunk enterprise and see if it comes with those 4k files that are missing. 
Is there a way or is it possible to delete the manifest file and have it repopulate? If I delete it and upgrade Splunk on the hvy frwrder, will that populate the manifest file with fresh new information?

[saranvishva]

Hi @Dallastek1, mrsparkle is responsible for web interface provided by the splunk.

[isoutamo]

Hi

the easiest way to fix this, is just install the same splunk version over old one on this host. Don't remove old before installation! This just update/add all needed files with fresh ones. All local modifications (I suppose that those are under local directories) are kept as they were now.

r. Ismo