After upgrading Search Head Cluster from 6.3.1 to 6.5.1, how do I resolve multiple "No such file or directory" errors?

[guimilare]

Hello Splunkers.

I've upgraded my Search Head Cluster (SHC) [6 members, 1 deploy] from version 6.3.1 to version 6.5.1 .
The upgrade of the deploy was OK.
However, after upgrading all the SHC members, we were not able to see to following:
- Job inspector: when clicking on the Job Inspector link of a search, I receive the error 404.
- View recent: when clicking on "View recent" on a saved search, I receive the error 404.
- SPL highlight: this is not working, the commands and arguments don't change color as they should.

It looks like Splunk Web did not change to 6.5.1.
The searches and dashboards are fine, I can search and report correctly.

On the deploy, I have the following results to the splunk validate files command:

/splunk_bin/splunk/bin/splunk validate files
        Validating installed files against hashes from '/splunk_bin/splunk/splunk-6.5.1-f74036626f0c-linux-2.6-x86_64-manifest'
        All installed files intact.

On any member of the SHC, I get the following message:

/splunk_bin/splunk/bin/splunk validate files
        Validating installed files against hashes from '/splunk_bin/splunk/splunk-6.5.1-f74036626f0c-linux-2.6-x86_64-manifest'
File '/splunk_bin/splunk/etc/apps/introspection_generator_addon/default/README' changed.
File '/splunk_bin/splunk/etc/apps/introspection_generator_addon/default/app.conf' changed.
File '/splunk_bin/splunk/etc/apps/sample_app/metadata/default.meta' changed.
File '/splunk_bin/splunk/etc/apps/search/bin/crawl_network.py' changed.
File '/splunk_bin/splunk/etc/apps/search/bin/erex.py' changed.
File '/splunk_bin/splunk/etc/apps/search/bin/predict.py' changed.
File '/splunk_bin/splunk/etc/apps/search/bin/runshellscript.py' changed.
File '/splunk_bin/splunk/etc/apps/search/default/data/ui/manager/admin_directory.prod_lite.xml' changed.
Could not open '/splunk_bin/splunk/etc/apps/search/default/data/ui/manager/admin_macros.prod_lite.xml': No such file or directory
File '/splunk_bin/splunk/etc/apps/search/default/data/ui/manager/admin_win-admon.xml' changed.
File '/splunk_bin/splunk/etc/apps/search/default/data/ui/manager/admin_win-event-log-collections.xml' changed.
...
Could not open '/splunk_bin/splunk/etc/apps/search/default/transforms.conf': No such file or directory
File '/splunk_bin/splunk/etc/apps/search/metadata/default.meta' changed.
File '/splunk_bin/splunk/etc/apps/user-prefs/default/app.conf' changed.
File '/splunk_bin/splunk/etc/apps/user-prefs/default/user-prefs.conf' changed.
File '/splunk_bin/splunk/etc/apps/user-prefs/metadata/default.meta' changed.

I get that some files were changed and some files do not exists.
I've tried to reinstall the version 6.5.1 and even tried to copy the files from deploy to SHC members. However, when starting Splunk, looks like it erases this files.

Have you guys ever saw this?
Any hints?

Regards,
Guilherme

[guimilare · Accepted solution]

I've found the problem...
The Search and Report default APP was in the shcluster folder in deploy for some reason.
After removing it and resending the bundle, the errors dissapeared.

Regards,

[woodcock]

Definitely open a support ticket ASAP. In the mean time, you can suppress the errors by doing the following:

cp /splunk_bin/splunk/splunk-6.5.1-f74036626f0c-linux-2.6-x86_64-manifest /splunk_bin/splunk/splunk-6.5.1-f74036626f0c-linux-2.6-x86_64-manifest.bak
cat /splunk_bin/splunk/splunk-6.5.1-f74036626f0c-linux-2.6-x86_64-manifest.bak |
grep -v "/splunk_bin/splunk/etc/apps/introspection_generator_addon/default/README" |
grep -v "/splunk_bin/splunk/etc/apps/introspection_generator_addon/default/app.conf" |
grep -v "/splunk_bin/splunk/etc/apps/sample_app/metadata/default.meta" |
grep -v "/splunk_bin/splunk/etc/apps/search/bin/crawl_network.py" |
grep -v "/splunk_bin/splunk/etc/apps/search/bin/erex.py" |
grep -v "/splunk_bin/splunk/etc/apps/search/bin/predict.py" |
grep -v "/splunk_bin/splunk/etc/apps/search/bin/runshellscript.py" |
grep -v "/splunk_bin/splunk/etc/apps/search/default/data/ui/manager/admin_directory.prod_lite.xml" |
grep -v "/splunk_bin/splunk/etc/apps/search/default/data/ui/manager/admin_macros.prod_lite.xml" |
grep -v "/splunk_bin/splunk/etc/apps/search/default/data/ui/manager/admin_win-admon.xml" |
grep -v "/splunk_bin/splunk/etc/apps/search/default/data/ui/manager/admin_win-event-log-collections.xml" |
grep -v "/splunk_bin/splunk/etc/apps/search/default/transforms.conf" |
grep -v "/splunk_bin/splunk/etc/apps/search/metadata/default.meta" |
grep -v "/splunk_bin/splunk/etc/apps/user-prefs/default/app.conf" |
grep -v "/splunk_bin/splunk/etc/apps/user-prefs/default/user-prefs.conf" |
grep -v "/splunk_bin/splunk/etc/apps/user-prefs/metadata/default.meta" >
/splunk_bin/splunk/splunk-6.5.1-f74036626f0c-linux-2.6-x86_64-manifest