Why am I receiving file integrity problems after upgrading from 6.4.1 to 6.5.2?

[horsefez]

Hi fellow Splunkers,

I recently upgraded my existing Splunk instances from 6.4.1 to 6.5.2.
My setup consists of an indexer and search head cluster and a master/deployer instance.
Before upgrading I read the documentation on what to keep in mind updating to 6.5.

The upgrade process completed without any errors. I then checked the newly implemented health check and the results returned a file integrity warning from our search heads.

The results look like this for every search head we have:

Looking into the actual files on the systems every file is there, even those who are reported missing.

What could be the cause of this?
Any help would be appreciated!

Regards,
pyro_wood

[woodcock · Accepted solution]

The file-integrity checks are accomplished by comparing the names, metadata, and checksums of every file that exists in the $SPLUNK_HOME/../splunk-<version>-<FooBarBlahJunk>-<platform>-manifest file for your installed Splunk version against files in $SPLUNK_HOME. You really should only have 1 manifest file in the $SPLUNK_HOME/.. directory, but you probably have many. Delete all of the files that DO NOT match your installed version. Also make sure that the file has permissions that will allow splunkd to read it. If problems persist, I would replace the manifest file with one from another identical server that is NOT having the problem.

[horsefez]

Hi woodcock,
thanks a lot!

I will try that, you are right... there is still the 6.4.1 manifest file.

[horsefez]

@woodcock

It kinda fixed it, but there is still one file that is reported to be different.

check_result            file_path
differs         /opt/splunk/etc/apps/search/default/app.conf

Could i edit the manifest file and simply delete the file-checksum at the specific entry?

f 444 splunk splunk splunk/etc/apps/search/default/app.conf 2ee60fc05d6f84344f97a96b2b7ddf166cf3da3626fde8c8b463e5a9470bf574

OR should I try to import a manifest file from another system?
OR should i try to import a app.conf file from another system?

[woodcock]

Well keep in mind what is at work. This is a feature of Splunk designed to let you know when people have been modifying files that should never be modified. In any case, you have multiple options, these of which are reasonable, IMHO:
o Reinstall
o Replace the file that has changed from a good copy from another server with same version/architecture

What you should probably not do (without very good reason) is modify the manifest file (although replacing it wholesale might be OK; I have never tried it).

[horsefez]

Thank you very much for the help so far.

So... I did replace the app.conf file with a file from another system and also i did so with the manifest file. Sadly no success here.

I looked into it a bit further and noticed something strange.
When I first imported the app.conf file the file reads like this:
# Version 6.5.2
#
# Splunk app configuration file
#

[install]
is_configured = true
state = enabled
allows_disable = false

[ui]
is_visible = true
label = Search & Reporting

[launcher]
author=Splunk
description=The Search app is Splunk's default interface for searching and analyzing IT data. It allows you to index data into Splunk, add knowledge, build reports, and create alerts. The Search app can be used across many areas of IT including application management, operations management, security, and compliance.
version=6.5.2

After I did a rolling-restart of the searchhead-cluster the app.conf file changes from permission 444 to 600 and there gets a line added to the [install] stanza that goes like this:

install_source_checksum = feac3d831bff2c53082ee20260d39deeb6de1b4d

Very strange behaviour. It only happens on the searchheads after the restart.
Maybe I have to reinstall them completely.

[woodcock]

I would report this is a bug in the SHC as regards integrity checking and then delete this file from your manifest file so that it will skip the check (or update it to match what is there now, if you think this change will only happen once, instead of a new change on every rolling restart). Reinstalling will change nothing.

[horsefez]

I will file this a bug and delete the line in the manifest.

Thank you!