Splunk Enterprise

File Integrity checks found files that did not match the system-provided manifest?

Gregski11
Contributor

Splunk version 9.0.0 on Windows servers

Please allow me to preface this by saying yes I GOOGLED this error and yes I did find some hits on this very Community site though 6 years old, and yes I did follow the many links to other links, which in turn lead me to more orphaned threads, ha ha, hence deciding to make this fresh more current post in which I hope we can present a solution

File Integrity checks found 40 files that did not match the system-provided manifest. Review the list of problems reported by the InstalledFileHashChecker in splunkd.log File Integrity Check View ; potentially restore files from installation media, change practices to avoid changing files, or work with support to identify the problem.

So if you click that link you get something like this:

 

List of installed files presenting integrity check failures

The table below shows files that were installed by the Splunk Enterprise package and have been improperly modified or are missing. Learn more.

 
Search is completed
 
File path Check result
 

List of installed files presenting integrity check failures
The table below shows files that were installed by the Splunk Enterprise package and have been improperly modified or are missing. Learn more.

Search is completed
File path Check result
C:\Program Files\Splunk\etc/apps/SplunkForwarder/default/app.conf missing
C:\Program Files\Splunk\etc/apps/SplunkForwarder/default/default-mode.conf missing
C:\Program Files\Splunk\etc/apps/SplunkForwarder/default/health.conf missing
C:\Program Files\Splunk\etc/apps/SplunkForwarder/default/outputs.conf missing
C:\Program Files\Splunk\etc/apps/SplunkForwarder/default/server.conf missing
C:\Program Files\Splunk\etc/apps/SplunkForwarder/metadata/default.meta missing
C:\Program Files\Splunk\etc/apps/python_upgrade_readiness_app/appserver/static/pages/jquery_scan.js differs
C:\Program Files\Splunk\etc/apps/python_upgrade_readiness_app/appserver/static/pages/python_scan.js differs
C:\Program Files\Splunk\etc/apps/python_upgrade_readiness_app/appserver/static/pages/setting.js differs
C:\Program Files\Splunk\etc/apps/python_upgrade_readiness_app/appserver/static/pages/setting_scan.js differs
C:\Program Files\Splunk\etc/apps/python_upgrade_readiness_app/appserver/static/pages/splunk9x_scan.js differs
C:\Program Files\Splunk\etc/apps/python_upgrade_readiness_app/bin/eura_app_list.py differs
C:\Program Files\Splunk\etc/apps/python_upgrade_readiness_app/bin/eura_check_mongodb_tls_dns_validation.py differs
C:\Program Files\Splunk\etc/apps/python_upgrade_readiness_app/bin/eura_check_search_peer_ssl_config.py differs
C:\Program Files\Splunk\etc/apps/python_upgrade_readiness_app/bin/eura_email_notification_switch_scripted_input.py differs
C:\Program Files\Splunk\etc/apps/python_upgrade_readiness_app/bin/eura_remote_latest_report.py differs
C:\Program Files\Splunk\etc/apps/python_upgrade_readiness_app/bin/eura_remote_scan_scripted_input.py differs
C:\Program Files\Splunk\etc/apps/python_upgrade_readiness_app/bin/eura_scan_apps.py differs
C:\Program Files\Splunk\etc/apps/python_upgrade_readiness_app/bin/eura_scan_deployment.py differs
C:\Program Files\Splunk\etc/apps/python_upgrade_readiness_app/bin/eura_scan_process.py differs
C:\Program Files\Splunk\etc/apps/python_upgrade_readiness_app/bin/eura_send_email.py differs
C:\Program Files\Splunk\etc/apps/python_upgrade_readiness_app/bin/eura_telemetry.py differs
C:\Program Files\Splunk\etc/apps/python_upgrade_readiness_app/bin/jura_scan_process.py differs
C:\Program Files\Splunk\etc/apps/python_upgrade_readiness_app/bin/jura_telemetry.py differs
C:\Program Files\Splunk\etc/apps/python_upgrade_readiness_app/bin/libs_py2/pura_libs_utils/pura_consts.py differs
C:\Program Files\Splunk\etc/apps/python_upgrade_readiness_app/bin/libs_py2/pura_libs_utils/pura_logger_manager.py differs
C:\Program Files\Splunk\etc/apps/python_upgrade_readiness_app/bin/libs_py2/pura_libs_utils/pura_skynet_log_manager.py differs
C:\Program Files\Splunk\etc/apps/python_upgrade_readiness_app/bin/libs_py2/pura_libs_utils/pura_utils.py differs
C:\Program Files\Splunk\etc/apps/python_upgrade_readiness_app/bin/libs_py2/pura_libs_utils/splunkbaseapps.csv differs
C:\Program Files\Splunk\etc/apps/python_upgrade_readiness_app/bin/libs_py3/pura_libs_utils/pura_consts.py differs
C:\Program Files\Splunk\etc/apps/python_upgrade_readiness_app/bin/libs_py3/pura_libs_utils/pura_logger_manager.py differs
C:\Program Files\Splunk\etc/apps/python_upgrade_readiness_app/bin/libs_py3/pura_libs_utils/pura_skynet_log_manager.py differs
C:\Program Files\Splunk\etc/apps/python_upgrade_readiness_app/bin/libs_py3/pura_libs_utils/pura_utils.py differs
C:\Program Files\Splunk\etc/apps/python_upgrade_readiness_app/bin/libs_py3/pura_libs_utils/splunkbaseapps.csv differs
C:\Program Files\Splunk\etc/apps/python_upgrade_readiness_app/bin/pura_remote_latest_report.py differs
C:\Program Files\Splunk\etc/apps/python_upgrade_readiness_app/bin/pura_telemetry.py differs
C:\Program Files\Splunk\etc/apps/python_upgrade_readiness_app/bin/scan_process.py differs
C:\Program Files\Splunk\etc/apps/python_upgrade_readiness_app/default/data/ui/nav/default.env_cloud.xml missing
C:\Program Files\Splunk\etc/apps/python_upgrade_readiness_app/default/data/ui/nav/default.xml differs
C:\Program Files\Splunk\etc/system/local/README missing

Labels (1)
0 Karma

spodda01da
Path Finder

Had similar issue when I upgraded the app from version 1.0.0 to 4.0.2. 

Reverted back to version 1.0.0 which seems to fix the issue.

0 Karma

Gregski11
Contributor

well this is rather peculiar after uninstalling the SplunkForwarder app properly through the Web UI, Splunk still looks for it and the missing subdirectory and all of it's contents still fail the File Integrity Check, this is so Bogus!

 

C:\Program Files\Splunk\etc/apps/SplunkForwarder/default/app.conf missing
C:\Program Files\Splunk\etc/apps/SplunkForwarder/default/default-mode.conf missing
C:\Program Files\Splunk\etc/apps/SplunkForwarder/default/health.conf missing
C:\Program Files\Splunk\etc/apps/SplunkForwarder/default/outputs.conf missing
C:\Program Files\Splunk\etc/apps/SplunkForwarder/default/server.conf missing
C:\Program Files\Splunk\etc/apps/SplunkForwarder/metadata/default.meta missing

 

So does this mean you can't delete this app on your Indexers for example? where the buck stops here 

0 Karma

Gregski11
Contributor

forgive me for the lengthy post, the intent is to help somebody out in the future, so after manually replacing more files from a sister server we have progress

File Integrity checks found 26 files that did not match the system-provided manifest. 

so we are down from 40 down to only 26

 

here's the batch of 10 more I just replaced

 

bin/libs_py2/pura_libs_utils/pura_consts.py
bin/libs_py2/pura_libs_utils/pura_logger_manager.py
bin/libs_py2/pura_libs_utils/pura_skynet_log_manager.py
bin/libs_py2/pura_libs_utils/pura_utils.py
bin/libs_py2/pura_libs_utils/splunkbaseapps.csv

bin/libs_py3/pura_libs_utils/pura_consts.py
bin/libs_py3/pura_libs_utils/pura_logger_manager.py
bin/libs_py3/pura_libs_utils/pura_skynet_log_manager.py
bin/libs_py3/pura_libs_utils/pura_utils.py
bin/libs_py3/pura_libs_utils/splunkbaseapps.csv

 

yes it's a pain, yes I had to copy them over manually and restart Splunk 

0 Karma

Gregski11
Contributor

well this is interesting and I think worth sharing

so I decided to move the python_upgrade_readiness_app to a new folder I created called Splunk\etc\apps we deleted in order to simulate deleting the app without actually deleting the app, made sense to me at the time

then I restarted Splunk, and whoa, it came back now saying we have 1,100 files that failed the integrity check, what on earth? talk about one step forward and a 1,000 steps back 

0 Karma

PickleRick
SplunkTrust
SplunkTrust

This simply means that some of the files that you are not supposed to touch were touched. That's it. What's the practical significance of this message? Depends on the files.

Just don't touch the things that come with splunk by default. If you want to disable app - disable it but don't just delete a whole folder.

0 Karma

Gregski11
Contributor

Rick I agree, the question is how do we fix it after someone else has touched them, it wasn't me it was the prior admin, this may be the scenario for other admins on here as well who inherit the environment 

0 Karma

PickleRick
SplunkTrust
SplunkTrust

The easiest way would be probably to get an installer, unpack it somewhere and copy over the missing files (you might need to fix ownership as well depending on what user your splunk is installed/runs under)

0 Karma

isoutamo
SplunkTrust
SplunkTrust

Unless someone have changed configurations on default folder the easiest way to fix this issue to just reinstall the same version again. Then it just add those missing files and also replace changed files on default directories. 

Anyhow in your situation I just take backup of SPLUNK_HOME/etc before do reinstallation, just for case that there has change something else. 

r. Ismo

0 Karma

Gregski11
Contributor

thank you all for trying to help and offering solutions, my manual fork lift replacement of those 5 files mentioned above did actually work (I just restarted Splunk on the wrong server at first, dope) 

my question now is why do I even need that app, can I just uninstall it and be done with it, and if and when we decide to upgrade Python we can just worry about reinstalling it then? 

0 Karma

Gregski11
Contributor

alright so next I decided to divide and conquer, as you can see a lot of the headache is attributed to the python_upgrade_readiness_app which I would deem non life threatening so I decided to fandangle with it a little bit starting with these 5 "static pages" files 

C:\Program Files\Splunk\etc/apps/python_upgrade_readiness_app/appserver/static/pages/jquery_scan.js
C:\Program Files\Splunk\etc/apps/python_upgrade_readiness_app/appserver/static/pages/python_scan.js
C:\Program Files\Splunk\etc/apps/python_upgrade_readiness_app/appserver/static/pages/setting.js
C:\Program Files\Splunk\etc/apps/python_upgrade_readiness_app/appserver/static/pages/setting_scan.js
C:\Program Files\Splunk\etc/apps/python_upgrade_readiness_app/appserver/static/pages/splunk9x_scan.js

I simply renamed them all to origina_file_name.js.differsand copied the identical files from a sister server with no intergrity check issues and restarted Splunk 

0 Karma

Gregski11
Contributor

so let's go after the low hanging fruit first:

C:\Program Files\Splunk\etc/system/local/README missing

 

the above README file is one that I myself indeed deleted, I simply did not want or need a REAME file cluttering up our /system/local subdirectories

I find it interesting that Splunk finds that a missing README file undermines the integrity of the system, it makes me question the validity of this alert in general, what are your thoughts?

 

0 Karma

Gregski11
Contributor

so it appears that clicking that link pretty much does the same thing as running this command

splunk validate files

0 Karma

Dallastek1
Path Finder

Ive got a little over 4k that show missing for S:\Program Files\Splunk\share/splunk/search_mrsparkle/xxx
on my heavy forwarder. Im still trying to determine what mrsparkle does, I may unpack a new splunk enterprise and see if it comes with those 4k files that are missing. 
Is there a way or is it possible to delete the manifest file and have it repopulate? If I delete it and upgrade Splunk on the hvy frwrder, will that populate the manifest file with fresh new information?

0 Karma

saranvishva
Explorer

Hi @Dallastek1mrsparkle is responsible for web interface provided by the splunk.

0 Karma

isoutamo
SplunkTrust
SplunkTrust

Hi

the easiest way to fix this, is just install the same splunk version over old one on this host. Don't remove old before installation! This just update/add all needed files with fresh ones. All local modifications (I suppose that those are under local directories) are kept as they were now.

r. Ismo

0 Karma
Get Updates on the Splunk Community!

.conf24 | Day 0

Hello Splunk Community! My name is Chris, and I'm based in Canberra, Australia's capital, and I travelled for ...

Enhance Security Visibility with Splunk Enterprise Security 7.1 through Threat ...

(view in My Videos)Struggling with alert fatigue, lack of context, and prioritization around security ...

Troubleshooting the OpenTelemetry Collector

  In this tech talk, you’ll learn how to troubleshoot the OpenTelemetry collector - from checking the ...