Splunk Enterprise

v9.1 upgrade issues - Why does Top tool bar shows Loading?

GrantMacRae
Explorer

Has anyone had this issue after upgrading to v9.1 from v 9.04, The top toolbar shows Loading....  but never loads?

The issue happened on our Search Head / Indexer & to the Deployment server.

I have also installed a full Splunk instance on to a virgin server & this has the same issue.

Edit - Ticket logged - CASE [3258488]

 

Splunk toolbar error.jpg

Labels (1)
Tags (3)

sini
Explorer

Hi,

Many thanks for posting.

This fixed the issue for us as well. It should however only be necessary to create a /etc/system/local/web.conf (or edit this one if already existing) and add:

#Workaround for toolbar not loading after 9.x upgrade
[settings]
minify_js = False

As always you should not modify anything in ./default in your Splunk instance unless you're an app developer creating an app.

Regards

0 Karma

dhatch
Explorer

Did a bit of testing on a different box, and correct, you do not need to adjust the "default" web.conf file.

I did have to edit the local web.conf as well as the "var/run/splunk/merged" web.conf files.  Just Local was not enough.

0 Karma

GerryUmbach
Loves-to-Learn Lots

We just upgraded today to v9.1.1 and are experiencing this "Loading" issue as well.

It's too bad that I didn't see this post, we would have delayed our upgrade in favour of another version.

Has anyone received guidance on the fix for this, or is everyone reverting back to a previous version?

0 Karma

dhatch
Explorer

Working with TAC, we figured out a solution for the Loading... issue.

There is an issue with the browser properly displaying the toolbar, this is mitigated by adjusting the web.conf file

There are a couple of places where I had to edit the file.

$Splunk_Home\etc\system\default\web.conf

$Splunk_Home\etc\system\local\web.conf

$Splunk_Home\var\run\splunk\merged\web.conf

You want to find the "minify_js" and set it to false.

minify_js = false

By default, it will be set to "true"

Restart the Splunk service, flush your browser cache and you should be good to go!

(if it does not work, there might be another web.conf file with the minify_js still set to true.  Search through and adjust as needed)

 

 

trashyroadz
Splunk Employee
Splunk Employee

Thank you for that, @dhatch !

This feature should be defaulted to "true" in $SPLUNK_HOME/etc/system/default/server.conf, starting with Splunk 9.1.x.

In previous versions, according to the web.conf.spec file, this attribute is not set at all.

Whenever making changes to .conf files in Splunk, please edit files from $SPLUNK_HOME/etc/system/local. If you do not yet have a web.conf file in that path when making such a change, create it, then include only the stanza(s) and attribute(s) you wish to modify.

Settings configured via $SPLUNK_HOME/etc/system/local take precedence over all other configuration files in all Splunk instances EXCEPT for clustered indexers. In the case of clustered indexers, the apps deployed to indexers by the Cluster Manager take precedence even over $SPLUNK_HOME/etc/system/local. 

See:

 

-- now that's Trashy!
0 Karma

JdeFalconr
Explorer

Just came here to say this worked for me, thanks for posting it.

0 Karma

GerryUmbach
Loves-to-Learn Lots

Thanks a bunch dhatch, we're back in business!

0 Karma

dhatch
Explorer

Glad to hear it!  Nice to know that the solution works on other systems as well.

0 Karma

akyz
Explorer

Only guidance I received was to downgrade however since Splunk doesn't support "downgrades" they provide no guidance. I ended up uninstalling and reinstalling 9.1, confirmed the issue was resolved after a clean install. Then proceeded to restore items/apps/config files 1 by 1.

0 Karma

zkrieger
Explorer

the ticket i had open was closed. you can generally work around the issue, if you can get the proper URL you need to do what you want to do and enter it directly. 

the issue is known and being repaired, but there is no notice of a resolution yet. 

 

0 Karma

dglass0215
Path Finder

I see they just released 9.1.1 today.  Wondering if anyone can confirm if this issue is fixed in 9.1.1?  Additionally if anyone who opened a case could respond, I'd like to hear what support is suggesting as I am currently experiencing this on some important screens.

Thanks,

David

0 Karma

dhatch
Explorer

I have a case open on it.  But all they are doing is suggesting clearing client caches and refreshing the server  by issuing the: http://<host>:<mport>/debug/refresh

None of that works at all.  I've tried different clients, different browsers, incognito mode, etc.  Nothing attempted makes any difference.

I really hope that someone figured it out and fixed it in the 9.1.1 code.

0 Karma

zkrieger
Explorer

Not sure why Splunk hasnt posted yet. This is with the development team now and being researched. No solution yet. If you have the issue, open a ticket so they can track it and notify you when a fix is available. 

0 Karma

dhatch
Explorer

Still happening on 9.1.0.2.  I finally got around to opening a case on it.

0 Karma

dhatch
Explorer

Any news on your case?  I am seeing the same thing on my 9.1.0.1 deployments.

0 Karma

GrantMacRae
Explorer

I eventually reverted back to 9.0.4, Running fine atm with no issues. I found that the 9.1.0.1 upgrade on top of the 9.1.0.0 didn't fix the issue. For me, I had the toolbar issue, the file integrity checks & the weirdest issue of all was that my indexing rate went through the roof. Increasing 5 fold.

Will wait till the stable version of 9.2 comes out.

 

GrantMacRae
Explorer

The fresh install of 9.1.0.1 worked perfectly, I rekon that there must be other issues with this update.

I'm going to roll back the search head / indexer & deployment server back to 9.0.4 & upgrade from there.

Glad I always keep full backups prior tho changing anything!

isoutamo
SplunkTrust
SplunkTrust

Good to have backups 😉

IMHO: If I was in your place, I will go to 9.0.5 and wait couple of weeks before go to 9.1.0.1 if possible 😉 Or even wait until 9.1.1 or 9.1.2. are published and tested by users.

0 Karma

isoutamo
SplunkTrust
SplunkTrust

Hi

I haven't seen this, but it seems that Splunk has withdrawn 9.1.0 version and currently the newest one is 9.0.5. 9.1.0 seems to have some other issues too and I think that there is no reason to upgrade it before it (or 9.1.0.1) will be available.

r. Ismo

isoutamo
SplunkTrust
SplunkTrust
9.1.0.1 has just released. Maybe it fix this? At least I didn't get anymore file integrity check warnings.
0 Karma
Get Updates on the Splunk Community!

Adoption of RUM and APM at Splunk

    Unleash the power of Splunk Observability   Watch Now In this can't miss Tech Talk! The Splunk Growth ...

March Community Office Hours Security Series Uncovered!

Hello Splunk Community! In March, Splunk Community Office Hours spotlighted our fabulous Splunk Threat ...

Stay Connected: Your Guide to April Tech Talks, Office Hours, and Webinars!

Take a look below to explore our upcoming Community Office Hours, Tech Talks, and Webinars in April. This post ...