Splunk upgrade from 7.3.3 to 8.0.6 fails with "WARNING: web interface does not seem to be available!"

[panulpet]

Hi,

Just can't get upgrade from  7.3.3 to 8.0.6 working.. Here I'm trying to upgrade clustered splunk (master node) and I also enabled some loggings in web.conf:

[settings
appServerProcessLogStderr = true

Then I could see at least following suspicious lines:

10-05-2020 19:46:46.301 +0300 WARN X509Verify - X509 certificate (O=SplunkUser,CN=SplunkServerDefaultCert) should not be used, as it is issued by Splunk's own default Certificate Authority (CA). This puts you
r Splunk instance at very high-risk of the MITM attack. Either commercial-CA-signed or self-CA-signed certificates must be used; see: <http://docs.splunk.com/Documentation/Splunk/latest/Security/Howtoself-sign
certificates>
10-05-2020 19:46:46.306 +0300 INFO TailReader - Registering metrics callback for: batchreader0
10-05-2020 19:46:46.306 +0300 INFO TailReader - Starting batchreader0 thread
10-05-2020 19:46:46.323 +0300 INFO UiHttpListener - Limiting UI HTTP server to 1365 sockets
10-05-2020 19:46:46.323 +0300 INFO UiHttpListener - Limiting UI HTTP server to 631 threads
10-05-2020 19:46:46.324 +0300 INFO UiAppServer - Starting stderr collecting thread
10-05-2020 19:46:46.626 +0300 INFO UiAppServer - From appserver: Traceback (most recent call last):
10-05-2020 19:46:46.626 +0300 INFO UiAppServer - From appserver: File "/opt/splunk/lib/python3.7/site-packages/splunk/appserver/mrsparkle/root.py", line 10, in <module>
10-05-2020 19:46:46.626 +0300 INFO UiAppServer - From appserver: from cherrypy import expose
10-05-2020 19:46:46.626 +0300 INFO UiAppServer - From appserver: File "/opt/splunk/lib/python3.7/site-packages/cherrypy/__init__.py", line 76, in <module>
10-05-2020 19:46:46.626 +0300 INFO UiAppServer - From appserver: from . import _cprequest, _cpserver, _cptree, _cplogging, _cpconfig
10-05-2020 19:46:46.626 +0300 INFO UiAppServer - From appserver: File "/opt/splunk/lib/python3.7/site-packages/cherrypy/_cpserver.py", line 6, in <module>
10-05-2020 19:46:46.626 +0300 INFO UiAppServer - From appserver: from cherrypy.process.servers import ServerAdapter
10-05-2020 19:46:46.626 +0300 INFO UiAppServer - From appserver: File "/opt/splunk/lib/python3.7/site-packages/cherrypy/process/__init__.py", line 13, in <module>
10-05-2020 19:46:46.626 +0300 INFO UiAppServer - From appserver: from .wspbus import bus
10-05-2020 19:46:46.626 +0300 INFO UiAppServer - From appserver: File "/opt/splunk/lib/python3.7/site-packages/cherrypy/process/wspbus.py", line 66, in <module>
10-05-2020 19:46:46.626 +0300 INFO UiAppServer - From appserver: import ctypes
10-05-2020 19:46:46.626 +0300 INFO UiAppServer - From appserver: File "/opt/splunk/lib/python3.7/ctypes/__init__.py", line 551, in <module>
10-05-2020 19:46:46.626 +0300 INFO UiAppServer - From appserver: _reset_cache()
10-05-2020 19:46:46.626 +0300 INFO UiAppServer - From appserver: File "/opt/splunk/lib/python3.7/ctypes/__init__.py", line 273, in _reset_cache
10-05-2020 19:46:46.626 +0300 INFO UiAppServer - From appserver: CFUNCTYPE(c_int)(lambda: None)
10-05-2020 19:46:46.626 +0300 INFO UiAppServer - From appserver: MemoryError

I can also deliver more logs if needed...

Any ideas what could be wrong?

[mookiie2005]

Just wanted to add that we experienced this issue as well with splunk 9.1.4 and 9.1.5.  Thanks for this info this was down for over a month for us!

[jking81]

Adding to this in case anyone else is having this issue.  It seems like when Python is executed something is attempting to write to /tmp which ends up with a memory error when /tmp is mounted with noexec.

Our solution was to add

TMPDIR=<writable path>

to splunk-launch.conf. 

[brettw]

This helped us find a hint of the problem in another environment - thank you for the post!

[chris_barrett]

Hi,

We experienced a similar issue when we recently upgraded our TEST environment from 7.3.7 to 8.0.6 - the AppServer and WebUI is working on the 3 clustered Search Heads but isn't working on any of the other servers (eg the Deployment Server).

We've discovered that one difference between the Search Heads and the other servers is that /tmp is mounted with the "noexec" flag on the "other" servers but is not set on the Search Heads.  When I remounted /tmp on the Deployment Server without the noexec flag the AppServer/WebUI started up.

I'm curious to know if your /tmp is set noexec and, if so, if remounting it without the noexec flag "fixes" the issue.

[panulpet]

Has memory requirements changed from Splunk version 7.3.3 in the case of Master Node? Now  when running 7.3.3:

MemTotal: 3880788 kB
MemFree: 995200 kB
MemAvailable: 3000696 kB

[panulpet]

Hi,

If there is some who could help I upload splunkd.log with debug logs.. Feel free to check it out 🙂

https://www.dropbox.com/s/5w7hnfx65bk3f4k/splunkd.log.gz?dl=0

[panulpet]

Now I have tried to upgrade Splunk 4-5 times and everytime same thing...  I have tried "Splunk Platform Upgrade Readiness App" but as I do not have any special apps in Master node that doesn't help anything..

I have tried to start splunk using --debug flag (splunk start --debug") but those millions of lines don't tell me where the problem is.

For me, this seems to some kind of bug in Splunk. The upgrade seems to be so simple that it's difficult to make a mistake there.

I have no clue what to do next?? 

[panulpet]

Hi,

I have now tried to upgrade 3 times and always the same result. I can't access Web UI. Can I attach logs somewhere?

This kind of erros seen in log file:

10-05-2020 19:39:59.015 +0300 ERROR UiPythonFallback - Couldn't start appserver process on port 8065: Appserver at http://127.0.0.1:8065 never started up. Set `appServerProcessLogStderr` to "true" under [settings] in web.conf. Restart,
try the operation again, and review splunkd.log for any messages that contain "UiAppServer - From appserver".
10-05-2020 19:39:59.015 +0300 ERROR UiPythonFallback - Couldn't start any appserver processes, UI will probably not function correctly!
10-05-2020 19:39:59.015 +0300 ERROR UiHttpListener - No app server is running, stop initializing http server
10-05-2020 19:40:56.949 +0300 WARN GetRemoteAuthToken - Unable to get auth token from peer: https://10.90.136.65:8089 due to: Connection refused

I I grep UiAppServer from splunkd.log:

grep UiAppServer splunkd.log
10-05-2020 19:39:59.015 +0300 ERROR UiPythonFallback - Couldn't start appserver process on port 8065: Appserver at http://127.0.0.1:8065 never started up. Set `appServerProcessLogStderr` to "true" under [settings] in web.conf. Restart, try the operation again, and review splunkd.log for any messages that contain "UiAppServer - From appserver".
10-05-2020 19:46:46.324 +0300 INFO UiAppServer - Starting stderr collecting thread
10-05-2020 19:46:46.626 +0300 INFO UiAppServer - From appserver: Traceback (most recent call last):
10-05-2020 19:46:46.626 +0300 INFO UiAppServer - From appserver: File "/opt/splunk/lib/python3.7/site-packages/splunk/appserver/mrsparkle/root.py", line 10, in <module>
10-05-2020 19:46:46.626 +0300 INFO UiAppServer - From appserver: from cherrypy import expose
10-05-2020 19:46:46.626 +0300 INFO UiAppServer - From appserver: File "/opt/splunk/lib/python3.7/site-packages/cherrypy/__init__.py", line 76, in <module>
10-05-2020 19:46:46.626 +0300 INFO UiAppServer - From appserver: from . import _cprequest, _cpserver, _cptree, _cplogging, _cpconfig
10-05-2020 19:46:46.626 +0300 INFO UiAppServer - From appserver: File "/opt/splunk/lib/python3.7/site-packages/cherrypy/_cpserver.py", line 6, in <module>
10-05-2020 19:46:46.626 +0300 INFO UiAppServer - From appserver: from cherrypy.process.servers import ServerAdapter
10-05-2020 19:46:46.626 +0300 INFO UiAppServer - From appserver: File "/opt/splunk/lib/python3.7/site-packages/cherrypy/process/__init__.py", line 13, in <module>
10-05-2020 19:46:46.626 +0300 INFO UiAppServer - From appserver: from .wspbus import bus
10-05-2020 19:46:46.626 +0300 INFO UiAppServer - From appserver: File "/opt/splunk/lib/python3.7/site-packages/cherrypy/process/wspbus.py", line 66, in <module>
10-05-2020 19:46:46.626 +0300 INFO UiAppServer - From appserver: import ctypes
10-05-2020 19:46:46.626 +0300 INFO UiAppServer - From appserver: File "/opt/splunk/lib/python3.7/ctypes/__init__.py", line 551, in <module>
10-05-2020 19:46:46.626 +0300 INFO UiAppServer - From appserver: _reset_cache()
10-05-2020 19:46:46.626 +0300 INFO UiAppServer - From appserver: File "/opt/splunk/lib/python3.7/ctypes/__init__.py", line 273, in _reset_cache
10-05-2020 19:46:46.626 +0300 INFO UiAppServer - From appserver: CFUNCTYPE(c_int)(lambda: None)
10-05-2020 19:46:46.626 +0300 INFO UiAppServer - From appserver: MemoryError

[richgalloway]

The logs provided are pretty standard stuff, don't indicate any problems that would prevent Splunk from starting, and don't include the error in the title of this posting.

It's not uncommon for the CLI to report "the web interface does not seem to be available" even though the UI *is* available.

Please revert your web.conf changes and try again.

[ewok]

Seeing this same issue doing an instance replacement with 9.0.4 or 9.0.5.  Was there ever a solution?

[chris_barrett]

Per my response from October 2020, we found the issue to lie with /tmp being flagged as being "noexec".  When we removed this and remount /tmp, the web service started up as expected.  My recollection is that it had something to do with "libffi".

[ewok]

Thank you Chris.  Do  you (or anyone else) know if this requirement is documented somewhere?  The implementation has security requirement requiring STIG compliance.  Having /tmp mounted exec (without noexec) is a finding.  So, to go with this it needs to be documented as a vendor dependency.  But I haven't found this in anything from Splunk as a requirement.