Deployment Architecture

After upgrading a search head cluster to Splunk 6.3, why are all our launcher app icons missing?

Lucas_K
Motivator

After upgrading to 6.3 (search head clustering) all our launcher app icons have disappeared even for default untouched apps.

ie. search & reporting.

alt text

The path to the icon shows an "undefined" path.

 <img src="/en-US/splunkd/__raw/servicesNS/undefined/search/static/appIcon.png">

Upgrade was performed by stopping all instances.

Upgrading deployer.
Restart deployer.
Upgrade all search head cluster member instances.
Start member instances.
Wait for 5 minutes.
Do a new deploy from deployer.

1 Solution

Ellen
Splunk Employee
Splunk Employee

As previously posted this is a known issue SPL-107449 where the UI is missing app icons and navigation drop downs.

It has been reported in 6.3 and 6.3.1

http://docs.splunk.com/Documentation/Splunk/6.3.1/ReleaseNotes/Knownissues

This is related from python 2.7.8 upgrade to python 2.7.9 where setting a cookie with brackets in the name can cause the issue.
Python has reported this bug in 2.7.9 and fixed in 2.7.10 as referenced in this link:
https://bugs.python.org/issue22931

Below are 2 options that can be used prior to the fix which is tentatively targeted for the next 6.3.x maintenance release.
Note: When the maintenance release is applied, it will overwrite this modified Cookie.py

Option 1:

1) Make a backup copy of $SPLUNK_HOME/lib/python2.7/Cookie.py

Keep in mind if for some reason you have custom changes in this Cookie.py, it will be overwritten by the replacement file for this fix in an upcoming maintenance release

2) Modify $SPLUNK_HOME/lib/python2.7/Cookie.py as the user who starts Splunk with the following changes:

From:

_LegalCharsPatt = r"[\w\d!#%&'~_`><@,:/\$\*\+\-\.\^\|\)\(\?\}\{\=]"

To:

_LegalCharsPatt = r"[\w\d!#%&'~_`><@,:/\$\*\+\-\.\^\|\)\(\?\[\]\}\{\=]"

3) Restart your Splunk search head(s)

Option 2:

1) Download the modified Cookie.py from here that includes the same fix as in Option 1
If you have issues with the download, please contact Support.

2) Make a backup copy of $SPLUNK_HOME/lib/python2.7/Cookie.py

3) Replace $SPLUNK_HOME/lib/python2.7/Cookie.py with the modified file

4) Restart your Splunk search head(s)

View solution in original post

Aditya_K_Rao
New Member

I have Splunk verision 6.4 and yet seeing this issue frequently on the SearchHead.
I tried the Python fix on all the Search Heads and restarted them but No Use. The icons and menu items are frozen until your lucky to have a working user experience. I even tried on all browsers and tried the incognito mode as well.

Any help is much appreciated.

0 Karma

jonathon
Path Finder

I saw this issue on a 6.2.3 Search Head Cluster (5 nodes).

I originally downloaded and distributed the Cookie.py but it didn't fix the issue, the problem is that the Cookie.py was a 6.3.x version instead of 6.2.3.

The fix for me was to apply the manual fix of changing _LegalCharsPatt.

After the edit, the diff results were as follows:

529,530c529
< #LegalCharsPatt = r"[\w\d!#%&'~`><@,:/\$*+-.^|)(\?}{=]"

< LegalCharsPatt = r"[\w\d!#%&'~`><@,:/\$*+-.^|)(\?[]}{=]"

>LegalCharsPatt = r"[\w\d!#%&'~`><@,:/\$*+-.^|)(\?}{=]"

distributing this to each cluster member and restarting each (captain last) corrected the issue for our environment.

0 Karma

apfender_splunk
Splunk Employee
Splunk Employee

You can give Firefox portable 41.0.2 a try, I never had such problem after upgrading SHC to 6.3.1 using that version. But I do see other people around using different versions of Firefox experiencing the missing icons and menu bar.

0 Karma

Ellen
Splunk Employee
Splunk Employee

As previously posted this is a known issue SPL-107449 where the UI is missing app icons and navigation drop downs.

It has been reported in 6.3 and 6.3.1

http://docs.splunk.com/Documentation/Splunk/6.3.1/ReleaseNotes/Knownissues

This is related from python 2.7.8 upgrade to python 2.7.9 where setting a cookie with brackets in the name can cause the issue.
Python has reported this bug in 2.7.9 and fixed in 2.7.10 as referenced in this link:
https://bugs.python.org/issue22931

Below are 2 options that can be used prior to the fix which is tentatively targeted for the next 6.3.x maintenance release.
Note: When the maintenance release is applied, it will overwrite this modified Cookie.py

Option 1:

1) Make a backup copy of $SPLUNK_HOME/lib/python2.7/Cookie.py

Keep in mind if for some reason you have custom changes in this Cookie.py, it will be overwritten by the replacement file for this fix in an upcoming maintenance release

2) Modify $SPLUNK_HOME/lib/python2.7/Cookie.py as the user who starts Splunk with the following changes:

From:

_LegalCharsPatt = r"[\w\d!#%&'~_`><@,:/\$\*\+\-\.\^\|\)\(\?\}\{\=]"

To:

_LegalCharsPatt = r"[\w\d!#%&'~_`><@,:/\$\*\+\-\.\^\|\)\(\?\[\]\}\{\=]"

3) Restart your Splunk search head(s)

Option 2:

1) Download the modified Cookie.py from here that includes the same fix as in Option 1
If you have issues with the download, please contact Support.

2) Make a backup copy of $SPLUNK_HOME/lib/python2.7/Cookie.py

3) Replace $SPLUNK_HOME/lib/python2.7/Cookie.py with the modified file

4) Restart your Splunk search head(s)

View solution in original post

arubi2
Explorer

Hi -

We are working on implementing this fix in our clustered environment (many, many search heads affected). Was curious if there's a specific set of repro instructions that we can confirm the fix with?

We are seeing the issue, but once cookies are cleared, the issue goes away, but for an indiscriminate amount of time and/or after a certain user interaction which we can't figure out easily.

Thanks!

0 Karma

john_dagostino
Path Finder

I hope this doesn't get buried, but for those still experiencing the issue support has come through with a new version of Cookie.py for those on 6.3.2 which has resolved the problem on all servers in my environment. It appears the only line which was modified from the 6.3.2 version is the same as the one mentioned above. If you're still experiencing the issue, please try the new code below in Cookie.py:

_LegalCharsPatt  = r"[\w\d!#%&'~_`><@,:/\$\*\+\-\.\^\|\)\(\?\}\{\=\[\]\"\\]"
0 Karma

matt_park
Explorer

For those still experiencing the issue, make sure the case matches your current file. The download version (Option 2) is lowercase and should be Cookie.py instead of cookie.py. I moved my original version to a backup directory and then copied in the updated version (with the proper case), restarted the search head and it resolved the issue.

As a temporary workaround, it seems using a private browsing session bypasses your cookies and should allow you to get a fully working interface.

john_dagostino
Path Finder

We recently upgraded from 6.2.4 to 6.3.2 and started experiencing this issue. Has anyone else confirmed that this is still not working in 6.3.2?

0 Karma

lquinn
Contributor

Hi John, I am also still experiencing the same issue in 6.3.2. Very frustrating!

john_dagostino
Path Finder

I've opened a support ticket, I'll post here if they offer a solution.

0 Karma

johnberwick
Splunk Employee
Splunk Employee

Can you edit the Cookie.Py and confirm that the

_LegalCharsPatt = r"[\w\d!#%&'~_`><@,:/\$\*\+\-\.\^\|\)\(\?\[\]\}\{\=]" is correct.

This should be resolved and would only think the could occur if the old Cookie.py is still there and wasn't updated.

Failing that Please open a ticket and provide a screen shot and Diag.

0 Karma

lquinn
Contributor

As mentioned in the comments above, I have tried everything to get this working. I edited the cookie.py myself when using 6.3.1 and it didn't work then. Since upgrading to 6.3.2 i have checked my cookie.py file and it is as above. Still not working. I am currently unable to open support tickets as my entitlement has expired for some reason. I am in the process of working with Splunk to get this reinstated.

0 Karma

john_dagostino
Path Finder

I assume something was lost in the formatting of your post, but I've confirmed that _LegalCharsPatt in Cookie.py matches the value posted by Ellen above.

0 Karma

Ellen
Splunk Employee
Splunk Employee

6.3.2 maintenance release is now available for download and has the corrected Cookie.py which fixes SPL-107449.

The one line change in the 6.3.2 Cookie.py is the same as the workaround provided.

lquinn
Contributor

This change has not fixed the issue for me ... fingers crossed for an update soon!

0 Karma

Lucas_K
Motivator

Did you restart after editing the file? I've done this on over 40 machines now and it definitely works on Linux 64 at least.

0 Karma

spsrasru
Path Finder

Is it working for you in SHC environment ?

0 Karma

lquinn
Contributor

I am working on a standalone for development ...

0 Karma

lquinn
Contributor

Yes, I've tried everything. I have checked and triple checked that the file has been changed, restarted several times. I even upgraded to 6.3.1 and did the whole process again but I got nothing ....

0 Karma

timpacl
Path Finder

The path doesn't seem to match what Splunk 6.3.1 implements on a fresh install. This work-around specifies $SPLUNK_HOME/lib/python2.7 but my installation has $SPLUNK_HOME/python2.7/lib. I found the file at this location, updated, restarted, and it was fixed.

0 Karma
Did you miss .conf21 Virtual?

Good news! The event's keynotes and many of its breakout sessions are now available online, and still totally FREE!