Why am I unable to search from my dashboard after upgrading Splunk?

[cwheeler33]

We upgraded from Splunk Ent 8.2.1 to Splunk Ent 9.0.4.1 installed on CentOS7

We have put all our dashboards in to "CompanyApp". Normally when I want to see details of an event I will click on "Open in Search" at the bottom of my dashboard. When I do a new tab is opened and normally it will show me the results. Now it just says "loading". I do have "admin" role assigned to myself.

I assume this is some kind of permissions issue, but where and what do I look for?
The 

[cwheeler33 · Accepted solution]

I had opened a support ticket and we found the problem. During the upgrade, looks like the search.xml was somehow edited/changed.

edit this file:
/opt/splunk/etc/apps/<MyApp>/local/data/ui/views/search.xml

the problem code: 

<?xml version="1.0"?>
<view template="pages/app.html" type="html" isDashboard="False">
<label>Search</label>
</view> 

this resolved the issue
#note - we replaced "apps.html" with "search.html". 

<?xml version="1.0"?>
<view template="pages/search.html" type="html" isDashboard="False">
<label>Search</label>
</view> 

after the change you have to restart your splunk services 

[isoutamo]

Hi

try to change all files to user splunk (or what ever user you are using for splunk at OS level.

Just like "chown -fR splunk:splunk /opt/splunk" 

r. Ismo

[cwheeler33]

Sorry, this did not help.

if it were at the OS level, I would have seen errors at the OS level. Specifically I would have seen it during that service start up. This seems to be a change with splunk itself, either a feature or a config setting.

[isoutamo]

You cannot see all those issues on OS levee when you start splunk as splunk don't try to read all files when it starts, it reads just those which it is needing at that time!

Many of splunk features are depending on files on disk. If it cannot read those files when needed then those features (e.g. some dashboards, menus etc.) are not available on splunk.

Have you found anything with ERROR, CRITICAL or WARN from splunk internal logs?