v9.1 upgrade issues - Why does Top tool bar shows Loading?

[GrantMacRae]

Has anyone had this issue after upgrading to v9.1 from v 9.04, The top toolbar shows Loading....  but never loads?

The issue happened on our Search Head / Indexer & to the Deployment server.

I have also installed a full Splunk instance on to a virgin server & this has the same issue.

Edit - Ticket logged - CASE [3258488]

[sini]

Hi,

Many thanks for posting.

This fixed the issue for us as well. It should however only be necessary to create a /etc/system/local/web.conf (or edit this one if already existing) and add:

#Workaround for toolbar not loading after 9.x upgrade
[settings]
minify_js = False

As always you should not modify anything in ./default in your Splunk instance unless you're an app developer creating an app.

Regards

[dhatch]

Did a bit of testing on a different box, and correct, you do not need to adjust the "default" web.conf file.

I did have to edit the local web.conf as well as the "var/run/splunk/merged" web.conf files.  Just Local was not enough.

[GerryUmbach]

We just upgraded today to v9.1.1 and are experiencing this "Loading" issue as well.

It's too bad that I didn't see this post, we would have delayed our upgrade in favour of another version.

Has anyone received guidance on the fix for this, or is everyone reverting back to a previous version?

[dhatch]

Working with TAC, we figured out a solution for the Loading... issue.

There is an issue with the browser properly displaying the toolbar, this is mitigated by adjusting the web.conf file

There are a couple of places where I had to edit the file.

$Splunk_Home\etc\system\default\web.conf

$Splunk_Home\etc\system\local\web.conf

$Splunk_Home\var\run\splunk\merged\web.conf

You want to find the "minify_js" and set it to false.

minify_js = false

By default, it will be set to "true"

Restart the Splunk service, flush your browser cache and you should be good to go!

(if it does not work, there might be another web.conf file with the minify_js still set to true.  Search through and adjust as needed)

[trashyroadz]

Thank you for that, @dhatch !

This feature should be defaulted to "true" in $SPLUNK_HOME/etc/system/default/server.conf, starting with Splunk 9.1.x.

In previous versions, according to the web.conf.spec file, this attribute is not set at all.

Whenever making changes to .conf files in Splunk, please edit files from $SPLUNK_HOME/etc/system/local. If you do not yet have a web.conf file in that path when making such a change, create it, then include only the stanza(s) and attribute(s) you wish to modify.

Settings configured via $SPLUNK_HOME/etc/system/local take precedence over all other configuration files in all Splunk instances EXCEPT for clustered indexers. In the case of clustered indexers, the apps deployed to indexers by the Cluster Manager take precedence even over $SPLUNK_HOME/etc/system/local. 

See:

• Splunk documentation on precedence
• This useful community article

[JdeFalconr]

Just came here to say this worked for me, thanks for posting it.

[GerryUmbach]

Thanks a bunch dhatch, we're back in business!

[dhatch]

Glad to hear it!  Nice to know that the solution works on other systems as well.

[akyz]

Only guidance I received was to downgrade however since Splunk doesn't support "downgrades" they provide no guidance. I ended up uninstalling and reinstalling 9.1, confirmed the issue was resolved after a clean install. Then proceeded to restore items/apps/config files 1 by 1.

[zkrieger]

the ticket i had open was closed. you can generally work around the issue, if you can get the proper URL you need to do what you want to do and enter it directly. 

the issue is known and being repaired, but there is no notice of a resolution yet. 

[dglass0215]

I see they just released 9.1.1 today.  Wondering if anyone can confirm if this issue is fixed in 9.1.1?  Additionally if anyone who opened a case could respond, I'd like to hear what support is suggesting as I am currently experiencing this on some important screens.

Thanks,

David

[dhatch]

I have a case open on it.  But all they are doing is suggesting clearing client caches and refreshing the server  by issuing the: http://<host>:<mport>/debug/refresh

None of that works at all.  I've tried different clients, different browsers, incognito mode, etc.  Nothing attempted makes any difference.

I really hope that someone figured it out and fixed it in the 9.1.1 code.

[zkrieger]

Not sure why Splunk hasnt posted yet. This is with the development team now and being researched. No solution yet. If you have the issue, open a ticket so they can track it and notify you when a fix is available. 

[dhatch]

Still happening on 9.1.0.2.  I finally got around to opening a case on it.

[dhatch]

Any news on your case?  I am seeing the same thing on my 9.1.0.1 deployments.

[GrantMacRae]

I eventually reverted back to 9.0.4, Running fine atm with no issues. I found that the 9.1.0.1 upgrade on top of the 9.1.0.0 didn't fix the issue. For me, I had the toolbar issue, the file integrity checks & the weirdest issue of all was that my indexing rate went through the roof. Increasing 5 fold.

Will wait till the stable version of 9.2 comes out.

[GrantMacRae]

The fresh install of 9.1.0.1 worked perfectly, I rekon that there must be other issues with this update.

I'm going to roll back the search head / indexer & deployment server back to 9.0.4 & upgrade from there.

Glad I always keep full backups prior tho changing anything!

[isoutamo]

Good to have backups 😉

IMHO: If I was in your place, I will go to 9.0.5 and wait couple of weeks before go to 9.1.0.1 if possible 😉 Or even wait until 9.1.1 or 9.1.2. are published and tested by users.

[isoutamo]

Hi

I haven't seen this, but it seems that Splunk has withdrawn 9.1.0 version and currently the newest one is 9.0.5. 9.1.0 seems to have some other issues too and I think that there is no reason to upgrade it before it (or 9.1.0.1) will be available.

r. Ismo

[isoutamo]

9.1.0.1 has just released. Maybe it fix this? At least I didn't get anymore file integrity check warnings.