Deployment Architecture

Why am I receiving file integrity problems after upgrading from 6.4.1 to 6.5.2?

horsefez
SplunkTrust
SplunkTrust

Hi fellow Splunkers,

I recently upgraded my existing Splunk instances from 6.4.1 to 6.5.2.
My setup consists of an indexer and search head cluster and a master/deployer instance.
Before upgrading I read the documentation on what to keep in mind updating to 6.5.

The upgrade process completed without any errors. I then checked the newly implemented health check and the results returned a file integrity warning from our search heads.

The results look like this for every search head we have:

alt text

Looking into the actual files on the systems every file is there, even those who are reported missing.

What could be the cause of this?
Any help would be appreciated!

Regards,
pyro_wood

0 Karma
1 Solution

woodcock
Esteemed Legend

The file-integrity checks are accomplished by comparing the names, metadata, and checksums of every file that exists in the $SPLUNK_HOME/../splunk-<version>-<FooBarBlahJunk>-<platform>-manifest file for your installed Splunk version against files in $SPLUNK_HOME. You really should only have 1 manifest file in the $SPLUNK_HOME/.. directory, but you probably have many. Delete all of the files that DO NOT match your installed version. Also make sure that the file has permissions that will allow splunkd to read it. If problems persist, I would replace the manifest file with one from another identical server that is NOT having the problem.

View solution in original post

0 Karma

woodcock
Esteemed Legend

The file-integrity checks are accomplished by comparing the names, metadata, and checksums of every file that exists in the $SPLUNK_HOME/../splunk-<version>-<FooBarBlahJunk>-<platform>-manifest file for your installed Splunk version against files in $SPLUNK_HOME. You really should only have 1 manifest file in the $SPLUNK_HOME/.. directory, but you probably have many. Delete all of the files that DO NOT match your installed version. Also make sure that the file has permissions that will allow splunkd to read it. If problems persist, I would replace the manifest file with one from another identical server that is NOT having the problem.

0 Karma

horsefez
SplunkTrust
SplunkTrust

Hi woodcock,
thanks a lot!

I will try that, you are right... there is still the 6.4.1 manifest file.

0 Karma

horsefez
SplunkTrust
SplunkTrust

@woodcock

It kinda fixed it, but there is still one file that is reported to be different.

check_result            file_path
differs         /opt/splunk/etc/apps/search/default/app.conf

Could i edit the manifest file and simply delete the file-checksum at the specific entry?

f 444 splunk splunk splunk/etc/apps/search/default/app.conf 2ee60fc05d6f84344f97a96b2b7ddf166cf3da3626fde8c8b463e5a9470bf574

OR should I try to import a manifest file from another system?
OR should i try to import a app.conf file from another system?

0 Karma

woodcock
Esteemed Legend

Well keep in mind what is at work. This is a feature of Splunk designed to let you know when people have been modifying files that should never be modified. In any case, you have multiple options, these of which are reasonable, IMHO:
o Reinstall
o Replace the file that has changed from a good copy from another server with same version/architecture

What you should probably not do (without very good reason) is modify the manifest file (although replacing it wholesale might be OK; I have never tried it).

0 Karma

horsefez
SplunkTrust
SplunkTrust

Thank you very much for the help so far.

So... I did replace the app.conf file with a file from another system and also i did so with the manifest file. Sadly no success here.

I looked into it a bit further and noticed something strange.
When I first imported the app.conf file the file reads like this:
# Version 6.5.2
#
# Splunk app configuration file
#

[install]
is_configured = true
state = enabled
allows_disable = false

[ui]
is_visible = true
label = Search & Reporting

[launcher]
author=Splunk
description=The Search app is Splunk's default interface for searching and analyzing IT data. It allows you to index data into Splunk, add knowledge, build reports, and create alerts. The Search app can be used across many areas of IT including application management, operations management, security, and compliance.
version=6.5.2

After I did a rolling-restart of the searchhead-cluster the app.conf file changes from permission 444 to 600 and there gets a line added to the [install] stanza that goes like this:

install_source_checksum = feac3d831bff2c53082ee20260d39deeb6de1b4d

Very strange behaviour. It only happens on the searchheads after the restart.
Maybe I have to reinstall them completely.

0 Karma

woodcock
Esteemed Legend

I would report this is a bug in the SHC as regards integrity checking and then delete this file from your manifest file so that it will skip the check (or update it to match what is there now, if you think this change will only happen once, instead of a new change on every rolling restart). Reinstalling will change nothing.

0 Karma

horsefez
SplunkTrust
SplunkTrust

I will file this a bug and delete the line in the manifest.

Thank you!

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...