Deployment Architecture

After upgrading Search Head Cluster from 6.3.1 to 6.5.1, how do I resolve multiple "No such file or directory" errors?

guimilare
Communicator

Hello Splunkers.

I've upgraded my Search Head Cluster (SHC) [6 members, 1 deploy] from version 6.3.1 to version 6.5.1 .
The upgrade of the deploy was OK.
However, after upgrading all the SHC members, we were not able to see to following:
- Job inspector: when clicking on the Job Inspector link of a search, I receive the error 404.
- View recent: when clicking on "View recent" on a saved search, I receive the error 404.
- SPL highlight: this is not working, the commands and arguments don't change color as they should.

It looks like Splunk Web did not change to 6.5.1.
The searches and dashboards are fine, I can search and report correctly.

On the deploy, I have the following results to the splunk validate files command:

/splunk_bin/splunk/bin/splunk validate files
        Validating installed files against hashes from '/splunk_bin/splunk/splunk-6.5.1-f74036626f0c-linux-2.6-x86_64-manifest'
        All installed files intact.

On any member of the SHC, I get the following message:

/splunk_bin/splunk/bin/splunk validate files
        Validating installed files against hashes from '/splunk_bin/splunk/splunk-6.5.1-f74036626f0c-linux-2.6-x86_64-manifest'
File '/splunk_bin/splunk/etc/apps/introspection_generator_addon/default/README' changed.
File '/splunk_bin/splunk/etc/apps/introspection_generator_addon/default/app.conf' changed.
File '/splunk_bin/splunk/etc/apps/sample_app/metadata/default.meta' changed.
File '/splunk_bin/splunk/etc/apps/search/bin/crawl_network.py' changed.
File '/splunk_bin/splunk/etc/apps/search/bin/erex.py' changed.
File '/splunk_bin/splunk/etc/apps/search/bin/predict.py' changed.
File '/splunk_bin/splunk/etc/apps/search/bin/runshellscript.py' changed.
File '/splunk_bin/splunk/etc/apps/search/default/data/ui/manager/admin_directory.prod_lite.xml' changed.
Could not open '/splunk_bin/splunk/etc/apps/search/default/data/ui/manager/admin_macros.prod_lite.xml': No such file or directory
File '/splunk_bin/splunk/etc/apps/search/default/data/ui/manager/admin_win-admon.xml' changed.
File '/splunk_bin/splunk/etc/apps/search/default/data/ui/manager/admin_win-event-log-collections.xml' changed.
...
Could not open '/splunk_bin/splunk/etc/apps/search/default/transforms.conf': No such file or directory
File '/splunk_bin/splunk/etc/apps/search/metadata/default.meta' changed.
File '/splunk_bin/splunk/etc/apps/user-prefs/default/app.conf' changed.
File '/splunk_bin/splunk/etc/apps/user-prefs/default/user-prefs.conf' changed.
File '/splunk_bin/splunk/etc/apps/user-prefs/metadata/default.meta' changed.

I get that some files were changed and some files do not exists.
I've tried to reinstall the version 6.5.1 and even tried to copy the files from deploy to SHC members. However, when starting Splunk, looks like it erases this files.

Have you guys ever saw this?
Any hints?

Regards,
Guilherme

1 Solution

guimilare
Communicator

I've found the problem...
The Search and Report default APP was in the shcluster folder in deploy for some reason.
After removing it and resending the bundle, the errors dissapeared.

Regards,

View solution in original post

0 Karma

guimilare
Communicator

I've found the problem...
The Search and Report default APP was in the shcluster folder in deploy for some reason.
After removing it and resending the bundle, the errors dissapeared.

Regards,

0 Karma

woodcock
Esteemed Legend

Definitely open a support ticket ASAP. In the mean time, you can suppress the errors by doing the following:

cp /splunk_bin/splunk/splunk-6.5.1-f74036626f0c-linux-2.6-x86_64-manifest /splunk_bin/splunk/splunk-6.5.1-f74036626f0c-linux-2.6-x86_64-manifest.bak
cat /splunk_bin/splunk/splunk-6.5.1-f74036626f0c-linux-2.6-x86_64-manifest.bak |
grep -v "/splunk_bin/splunk/etc/apps/introspection_generator_addon/default/README" |
grep -v "/splunk_bin/splunk/etc/apps/introspection_generator_addon/default/app.conf" |
grep -v "/splunk_bin/splunk/etc/apps/sample_app/metadata/default.meta" |
grep -v "/splunk_bin/splunk/etc/apps/search/bin/crawl_network.py" |
grep -v "/splunk_bin/splunk/etc/apps/search/bin/erex.py" |
grep -v "/splunk_bin/splunk/etc/apps/search/bin/predict.py" |
grep -v "/splunk_bin/splunk/etc/apps/search/bin/runshellscript.py" |
grep -v "/splunk_bin/splunk/etc/apps/search/default/data/ui/manager/admin_directory.prod_lite.xml" |
grep -v "/splunk_bin/splunk/etc/apps/search/default/data/ui/manager/admin_macros.prod_lite.xml" |
grep -v "/splunk_bin/splunk/etc/apps/search/default/data/ui/manager/admin_win-admon.xml" |
grep -v "/splunk_bin/splunk/etc/apps/search/default/data/ui/manager/admin_win-event-log-collections.xml" |
grep -v "/splunk_bin/splunk/etc/apps/search/default/transforms.conf" |
grep -v "/splunk_bin/splunk/etc/apps/search/metadata/default.meta" |
grep -v "/splunk_bin/splunk/etc/apps/user-prefs/default/app.conf" |
grep -v "/splunk_bin/splunk/etc/apps/user-prefs/default/user-prefs.conf" |
grep -v "/splunk_bin/splunk/etc/apps/user-prefs/metadata/default.meta" >
/splunk_bin/splunk/splunk-6.5.1-f74036626f0c-linux-2.6-x86_64-manifest
0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...