After upgrading a search head cluster to Splunk 6.3, why are all our launcher app icons missing?

[Lucas_K]

After upgrading to 6.3 (search head clustering) all our launcher app icons have disappeared even for default untouched apps.

ie. search & reporting.

The path to the icon shows an "undefined" path.

 <img src="/en-US/splunkd/__raw/servicesNS/undefined/search/static/appIcon.png">

Upgrade was performed by stopping all instances.

Upgrading deployer.
Restart deployer.
Upgrade all search head cluster member instances.
Start member instances.
Wait for 5 minutes.
Do a new deploy from deployer.

[Aditya_K_Rao]

I have Splunk verision 6.4 and yet seeing this issue frequently on the SearchHead.
I tried the Python fix on all the Search Heads and restarted them but No Use. The icons and menu items are frozen until your lucky to have a working user experience. I even tried on all browsers and tried the incognito mode as well.

Any help is much appreciated.

[jonathon]

I saw this issue on a 6.2.3 Search Head Cluster (5 nodes).

I originally downloaded and distributed the Cookie.py but it didn't fix the issue, the problem is that the Cookie.py was a 6.3.x version instead of 6.2.3.

The fix for me was to apply the manual fix of changing _LegalCharsPatt.

After the edit, the diff results were as follows:

529,530c529
< #LegalCharsPatt = r"[\w\d!#%&'~`><@,:/\$*+-.^|)(\?}{=]"

< LegalCharsPatt = r"[\w\d!#%&'~`><@,:/\$*+-.^|)(\?[]}{=]"

>LegalCharsPatt = r"[\w\d!#%&'~`><@,:/\$*+-.^|)(\?}{=]"

distributing this to each cluster member and restarting each (captain last) corrected the issue for our environment.

[apfender_splunk]

You can give Firefox portable 41.0.2 a try, I never had such problem after upgrading SHC to 6.3.1 using that version. But I do see other people around using different versions of Firefox experiencing the missing icons and menu bar.

[Ellen · Accepted solution]

As previously posted this is a known issue SPL-107449 where the UI is missing app icons and navigation drop downs.

It has been reported in 6.3 and 6.3.1

http://docs.splunk.com/Documentation/Splunk/6.3.1/ReleaseNotes/Knownissues

This is related from python 2.7.8 upgrade to python 2.7.9 where setting a cookie with brackets in the name can cause the issue.
Python has reported this bug in 2.7.9 and fixed in 2.7.10 as referenced in this link:
https://bugs.python.org/issue22931

Below are 2 options that can be used prior to the fix which is tentatively targeted for the next 6.3.x maintenance release.
Note: When the maintenance release is applied, it will overwrite this modified Cookie.py

Option 1:

1) Make a backup copy of $SPLUNK_HOME/lib/python2.7/Cookie.py

Keep in mind if for some reason you have custom changes in this Cookie.py, it will be overwritten by the replacement file for this fix in an upcoming maintenance release

2) Modify $SPLUNK_HOME/lib/python2.7/Cookie.py as the user who starts Splunk with the following changes:

From:

_LegalCharsPatt = r"[\w\d!#%&'~_`><@,:/\$\*\+\-\.\^\|\)\(\?\}\{\=]"

To:

_LegalCharsPatt = r"[\w\d!#%&'~_`><@,:/\$\*\+\-\.\^\|\)\(\?\[\]\}\{\=]"

3) Restart your Splunk search head(s)

Option 2:

1) Download the modified Cookie.py from here that includes the same fix as in Option 1
If you have issues with the download, please contact Support.

2) Make a backup copy of $SPLUNK_HOME/lib/python2.7/Cookie.py

3) Replace $SPLUNK_HOME/lib/python2.7/Cookie.py with the modified file

4) Restart your Splunk search head(s)

[arubi2]

Hi -

We are working on implementing this fix in our clustered environment (many, many search heads affected). Was curious if there's a specific set of repro instructions that we can confirm the fix with?

We are seeing the issue, but once cookies are cleared, the issue goes away, but for an indiscriminate amount of time and/or after a certain user interaction which we can't figure out easily.

Thanks!

[john_dagostino]

I hope this doesn't get buried, but for those still experiencing the issue support has come through with a new version of Cookie.py for those on 6.3.2 which has resolved the problem on all servers in my environment. It appears the only line which was modified from the 6.3.2 version is the same as the one mentioned above. If you're still experiencing the issue, please try the new code below in Cookie.py:

_LegalCharsPatt  = r"[\w\d!#%&'~_`><@,:/\$\*\+\-\.\^\|\)\(\?\}\{\=\[\]\"\\]"

[matt_park]

For those still experiencing the issue, make sure the case matches your current file. The download version (Option 2) is lowercase and should be Cookie.py instead of cookie.py. I moved my original version to a backup directory and then copied in the updated version (with the proper case), restarted the search head and it resolved the issue.

As a temporary workaround, it seems using a private browsing session bypasses your cookies and should allow you to get a fully working interface.

[john_dagostino]

We recently upgraded from 6.2.4 to 6.3.2 and started experiencing this issue. Has anyone else confirmed that this is still not working in 6.3.2?

[lquinn]

Hi John, I am also still experiencing the same issue in 6.3.2. Very frustrating!

[john_dagostino]

I've opened a support ticket, I'll post here if they offer a solution.

[johnberwick]

Can you edit the Cookie.Py and confirm that the

_LegalCharsPatt = r"[\w\d!#%&'~_`><@,:/\$\*\+\-\.\^\|\)\(\?\[\]\}\{\=]" is correct.

This should be resolved and would only think the could occur if the old Cookie.py is still there and wasn't updated.

Failing that Please open a ticket and provide a screen shot and Diag.

[lquinn]

As mentioned in the comments above, I have tried everything to get this working. I edited the cookie.py myself when using 6.3.1 and it didn't work then. Since upgrading to 6.3.2 i have checked my cookie.py file and it is as above. Still not working. I am currently unable to open support tickets as my entitlement has expired for some reason. I am in the process of working with Splunk to get this reinstated.

[john_dagostino]

I assume something was lost in the formatting of your post, but I've confirmed that _LegalCharsPatt in Cookie.py matches the value posted by Ellen above.

[Ellen]

6.3.2 maintenance release is now available for download and has the corrected Cookie.py which fixes SPL-107449.

The one line change in the 6.3.2 Cookie.py is the same as the workaround provided.

[lquinn]

This change has not fixed the issue for me ... fingers crossed for an update soon!

[Lucas_K]

Did you restart after editing the file? I've done this on over 40 machines now and it definitely works on Linux 64 at least.

[spsrasru]

Is it working for you in SHC environment ?

[lquinn]

I am working on a standalone for development ...

[lquinn]

Yes, I've tried everything. I have checked and triple checked that the file has been changed, restarted several times. I even upgraded to 6.3.1 and did the whole process again but I got nothing ....

[timpacl]

The path doesn't seem to match what Splunk 6.3.1 implements on a fresh install. This work-around specifies $SPLUNK_HOME/lib/python2.7 but my installation has $SPLUNK_HOME/python2.7/lib. I found the file at this location, updated, restarted, and it was fixed.