Installation

Splunk upgrade from 7.3.3 to 8.0.6 fails with "WARNING: web interface does not seem to be available!"

panulpet
Loves-to-Learn

Hi,

Just can't get upgrade from  7.3.3 to 8.0.6 working.. Here I'm trying to upgrade clustered splunk (master node) and I also enabled some loggings in web.conf:

[settings
appServerProcessLogStderr = true

Then I could see at least following suspicious lines:

10-05-2020 19:46:46.301 +0300 WARN X509Verify - X509 certificate (O=SplunkUser,CN=SplunkServerDefaultCert) should not be used, as it is issued by Splunk's own default Certificate Authority (CA). This puts you
r Splunk instance at very high-risk of the MITM attack. Either commercial-CA-signed or self-CA-signed certificates must be used; see: <http://docs.splunk.com/Documentation/Splunk/latest/Security/Howtoself-sign
certificates>
10-05-2020 19:46:46.306 +0300 INFO TailReader - Registering metrics callback for: batchreader0
10-05-2020 19:46:46.306 +0300 INFO TailReader - Starting batchreader0 thread
10-05-2020 19:46:46.323 +0300 INFO UiHttpListener - Limiting UI HTTP server to 1365 sockets
10-05-2020 19:46:46.323 +0300 INFO UiHttpListener - Limiting UI HTTP server to 631 threads
10-05-2020 19:46:46.324 +0300 INFO UiAppServer - Starting stderr collecting thread
10-05-2020 19:46:46.626 +0300 INFO UiAppServer - From appserver: Traceback (most recent call last):
10-05-2020 19:46:46.626 +0300 INFO UiAppServer - From appserver: File "/opt/splunk/lib/python3.7/site-packages/splunk/appserver/mrsparkle/root.py", line 10, in <module>
10-05-2020 19:46:46.626 +0300 INFO UiAppServer - From appserver: from cherrypy import expose
10-05-2020 19:46:46.626 +0300 INFO UiAppServer - From appserver: File "/opt/splunk/lib/python3.7/site-packages/cherrypy/__init__.py", line 76, in <module>
10-05-2020 19:46:46.626 +0300 INFO UiAppServer - From appserver: from . import _cprequest, _cpserver, _cptree, _cplogging, _cpconfig
10-05-2020 19:46:46.626 +0300 INFO UiAppServer - From appserver: File "/opt/splunk/lib/python3.7/site-packages/cherrypy/_cpserver.py", line 6, in <module>
10-05-2020 19:46:46.626 +0300 INFO UiAppServer - From appserver: from cherrypy.process.servers import ServerAdapter
10-05-2020 19:46:46.626 +0300 INFO UiAppServer - From appserver: File "/opt/splunk/lib/python3.7/site-packages/cherrypy/process/__init__.py", line 13, in <module>
10-05-2020 19:46:46.626 +0300 INFO UiAppServer - From appserver: from .wspbus import bus
10-05-2020 19:46:46.626 +0300 INFO UiAppServer - From appserver: File "/opt/splunk/lib/python3.7/site-packages/cherrypy/process/wspbus.py", line 66, in <module>
10-05-2020 19:46:46.626 +0300 INFO UiAppServer - From appserver: import ctypes
10-05-2020 19:46:46.626 +0300 INFO UiAppServer - From appserver: File "/opt/splunk/lib/python3.7/ctypes/__init__.py", line 551, in <module>
10-05-2020 19:46:46.626 +0300 INFO UiAppServer - From appserver: _reset_cache()
10-05-2020 19:46:46.626 +0300 INFO UiAppServer - From appserver: File "/opt/splunk/lib/python3.7/ctypes/__init__.py", line 273, in _reset_cache
10-05-2020 19:46:46.626 +0300 INFO UiAppServer - From appserver: CFUNCTYPE(c_int)(lambda: None)
10-05-2020 19:46:46.626 +0300 INFO UiAppServer - From appserver: MemoryError

I can also deliver more logs if needed...

Any ideas what could be wrong?

Labels (1)
0 Karma

chris_barrett
Communicator

Hi,

We experienced a similar issue when we recently upgraded our TEST environment from 7.3.7 to 8.0.6 - the AppServer and WebUI is working on the 3 clustered Search Heads but isn't working on any of the other servers (eg the Deployment Server).

We've discovered that one difference between the Search Heads and the other servers is that /tmp is mounted with the "noexec" flag on the "other" servers but is not set on the Search Heads.  When I remounted /tmp on the Deployment Server without the noexec flag the AppServer/WebUI started up.

I'm curious to know if your /tmp is set noexec and, if so, if remounting it without the noexec flag "fixes" the issue.

panulpet
Loves-to-Learn

Has memory requirements changed from Splunk version 7.3.3 in the case of Master Node? Now  when running 7.3.3:

MemTotal: 3880788 kB
MemFree: 995200 kB
MemAvailable: 3000696 kB

 

0 Karma

panulpet
Loves-to-Learn

Hi,

 

If there is some who could help I upload splunkd.log with debug logs.. Feel free to check it out 🙂

 

https://www.dropbox.com/s/5w7hnfx65bk3f4k/splunkd.log.gz?dl=0

0 Karma

panulpet
Loves-to-Learn

Now I have tried to upgrade Splunk 4-5 times and everytime same thing...  I have tried "Splunk Platform Upgrade Readiness App" but as I do not have any special apps in Master node that doesn't help anything..

I have tried to start splunk using --debug flag (splunk start --debug") but those millions of lines don't tell me where the problem is.

For me, this seems to some kind of bug in Splunk. The upgrade seems to be so simple that it's difficult to make a mistake there.

I have no clue what to do next?? 

 

0 Karma

panulpet
Loves-to-Learn

Hi,

 

I have now tried to upgrade 3 times and always the same result. I can't access Web UI. Can I attach logs somewhere?

 

This kind of erros seen in log file:

 

10-05-2020 19:39:59.015 +0300 ERROR UiPythonFallback - Couldn't start appserver process on port 8065: Appserver at http://127.0.0.1:8065 never started up. Set `appServerProcessLogStderr` to "true" under [settings] in web.conf. Restart,
try the operation again, and review splunkd.log for any messages that contain "UiAppServer - From appserver".
10-05-2020 19:39:59.015 +0300 ERROR UiPythonFallback - Couldn't start any appserver processes, UI will probably not function correctly!
10-05-2020 19:39:59.015 +0300 ERROR UiHttpListener - No app server is running, stop initializing http server
10-05-2020 19:40:56.949 +0300 WARN GetRemoteAuthToken - Unable to get auth token from peer: https://10.90.136.65:8089 due to: Connection refused

 

I I grep UiAppServer from splunkd.log:

 

grep UiAppServer splunkd.log
10-05-2020 19:39:59.015 +0300 ERROR UiPythonFallback - Couldn't start appserver process on port 8065: Appserver at http://127.0.0.1:8065 never started up. Set `appServerProcessLogStderr` to "true" under [settings] in web.conf. Restart, try the operation again, and review splunkd.log for any messages that contain "UiAppServer - From appserver".
10-05-2020 19:46:46.324 +0300 INFO UiAppServer - Starting stderr collecting thread
10-05-2020 19:46:46.626 +0300 INFO UiAppServer - From appserver: Traceback (most recent call last):
10-05-2020 19:46:46.626 +0300 INFO UiAppServer - From appserver: File "/opt/splunk/lib/python3.7/site-packages/splunk/appserver/mrsparkle/root.py", line 10, in <module>
10-05-2020 19:46:46.626 +0300 INFO UiAppServer - From appserver: from cherrypy import expose
10-05-2020 19:46:46.626 +0300 INFO UiAppServer - From appserver: File "/opt/splunk/lib/python3.7/site-packages/cherrypy/__init__.py", line 76, in <module>
10-05-2020 19:46:46.626 +0300 INFO UiAppServer - From appserver: from . import _cprequest, _cpserver, _cptree, _cplogging, _cpconfig
10-05-2020 19:46:46.626 +0300 INFO UiAppServer - From appserver: File "/opt/splunk/lib/python3.7/site-packages/cherrypy/_cpserver.py", line 6, in <module>
10-05-2020 19:46:46.626 +0300 INFO UiAppServer - From appserver: from cherrypy.process.servers import ServerAdapter
10-05-2020 19:46:46.626 +0300 INFO UiAppServer - From appserver: File "/opt/splunk/lib/python3.7/site-packages/cherrypy/process/__init__.py", line 13, in <module>
10-05-2020 19:46:46.626 +0300 INFO UiAppServer - From appserver: from .wspbus import bus
10-05-2020 19:46:46.626 +0300 INFO UiAppServer - From appserver: File "/opt/splunk/lib/python3.7/site-packages/cherrypy/process/wspbus.py", line 66, in <module>
10-05-2020 19:46:46.626 +0300 INFO UiAppServer - From appserver: import ctypes
10-05-2020 19:46:46.626 +0300 INFO UiAppServer - From appserver: File "/opt/splunk/lib/python3.7/ctypes/__init__.py", line 551, in <module>
10-05-2020 19:46:46.626 +0300 INFO UiAppServer - From appserver: _reset_cache()
10-05-2020 19:46:46.626 +0300 INFO UiAppServer - From appserver: File "/opt/splunk/lib/python3.7/ctypes/__init__.py", line 273, in _reset_cache
10-05-2020 19:46:46.626 +0300 INFO UiAppServer - From appserver: CFUNCTYPE(c_int)(lambda: None)
10-05-2020 19:46:46.626 +0300 INFO UiAppServer - From appserver: MemoryError

 

 

 

0 Karma

richgalloway
SplunkTrust
SplunkTrust

The logs provided are pretty standard stuff, don't indicate any problems that would prevent Splunk from starting, and don't include the error in the title of this posting.

It's not uncommon for the CLI to report "the web interface does not seem to be available" even though the UI *is* available.

Please revert your web.conf changes and try again.

---
If this reply helps you, Karma would be appreciated.
0 Karma

ewok
Observer

Seeing this same issue doing an instance replacement with 9.0.4 or 9.0.5.  Was there ever a solution?

0 Karma

chris_barrett
Communicator

Per my response from October 2020, we found the issue to lie with /tmp being flagged as being "noexec".  When we removed this and remount /tmp, the web service started up as expected.  My recollection is that it had something to do with "libffi".

 

0 Karma

ewok
Observer

Thank you Chris.  Do  you (or anyone else) know if this requirement is documented somewhere?  The implementation has security requirement requiring STIG compliance.  Having /tmp mounted exec (without noexec) is a finding.  So, to go with this it needs to be documented as a vendor dependency.  But I haven't found this in anything from Splunk as a requirement.

0 Karma
Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

Using the Splunk Threat Research Team’s Latest Security Content

REGISTER HERE Tech Talk | Security Edition Did you know the Splunk Threat Research Team regularly releases ...

SplunkTrust | 2024 SplunkTrust Application Period is Open!

It's that time again, folks! That's right, the application/nomination period for the 2024 SplunkTrust is ...