Installation

Splunk upgrade from 7.3.3 to 8.0.6 fails with "WARNING: web interface does not seem to be available!"

panulpet
Loves-to-Learn

Hi,

Just can't get upgrade from  7.3.3 to 8.0.6 working.. Here I'm trying to upgrade clustered splunk (master node) and I also enabled some loggings in web.conf:

[settings
appServerProcessLogStderr = true

Then I could see at least following suspicious lines:

10-05-2020 19:46:46.301 +0300 WARN X509Verify - X509 certificate (O=SplunkUser,CN=SplunkServerDefaultCert) should not be used, as it is issued by Splunk's own default Certificate Authority (CA). This puts you
r Splunk instance at very high-risk of the MITM attack. Either commercial-CA-signed or self-CA-signed certificates must be used; see: <http://docs.splunk.com/Documentation/Splunk/latest/Security/Howtoself-sign
certificates>
10-05-2020 19:46:46.306 +0300 INFO TailReader - Registering metrics callback for: batchreader0
10-05-2020 19:46:46.306 +0300 INFO TailReader - Starting batchreader0 thread
10-05-2020 19:46:46.323 +0300 INFO UiHttpListener - Limiting UI HTTP server to 1365 sockets
10-05-2020 19:46:46.323 +0300 INFO UiHttpListener - Limiting UI HTTP server to 631 threads
10-05-2020 19:46:46.324 +0300 INFO UiAppServer - Starting stderr collecting thread
10-05-2020 19:46:46.626 +0300 INFO UiAppServer - From appserver: Traceback (most recent call last):
10-05-2020 19:46:46.626 +0300 INFO UiAppServer - From appserver: File "/opt/splunk/lib/python3.7/site-packages/splunk/appserver/mrsparkle/root.py", line 10, in <module>
10-05-2020 19:46:46.626 +0300 INFO UiAppServer - From appserver: from cherrypy import expose
10-05-2020 19:46:46.626 +0300 INFO UiAppServer - From appserver: File "/opt/splunk/lib/python3.7/site-packages/cherrypy/__init__.py", line 76, in <module>
10-05-2020 19:46:46.626 +0300 INFO UiAppServer - From appserver: from . import _cprequest, _cpserver, _cptree, _cplogging, _cpconfig
10-05-2020 19:46:46.626 +0300 INFO UiAppServer - From appserver: File "/opt/splunk/lib/python3.7/site-packages/cherrypy/_cpserver.py", line 6, in <module>
10-05-2020 19:46:46.626 +0300 INFO UiAppServer - From appserver: from cherrypy.process.servers import ServerAdapter
10-05-2020 19:46:46.626 +0300 INFO UiAppServer - From appserver: File "/opt/splunk/lib/python3.7/site-packages/cherrypy/process/__init__.py", line 13, in <module>
10-05-2020 19:46:46.626 +0300 INFO UiAppServer - From appserver: from .wspbus import bus
10-05-2020 19:46:46.626 +0300 INFO UiAppServer - From appserver: File "/opt/splunk/lib/python3.7/site-packages/cherrypy/process/wspbus.py", line 66, in <module>
10-05-2020 19:46:46.626 +0300 INFO UiAppServer - From appserver: import ctypes
10-05-2020 19:46:46.626 +0300 INFO UiAppServer - From appserver: File "/opt/splunk/lib/python3.7/ctypes/__init__.py", line 551, in <module>
10-05-2020 19:46:46.626 +0300 INFO UiAppServer - From appserver: _reset_cache()
10-05-2020 19:46:46.626 +0300 INFO UiAppServer - From appserver: File "/opt/splunk/lib/python3.7/ctypes/__init__.py", line 273, in _reset_cache
10-05-2020 19:46:46.626 +0300 INFO UiAppServer - From appserver: CFUNCTYPE(c_int)(lambda: None)
10-05-2020 19:46:46.626 +0300 INFO UiAppServer - From appserver: MemoryError

I can also deliver more logs if needed...

Any ideas what could be wrong?

Labels (1)
0 Karma

mookiie2005
Communicator

Just wanted to add that we experienced this issue as well with splunk 9.1.4 and 9.1.5.  Thanks for this info this was down for over a month for us!

0 Karma

jking81
Explorer

Adding to this in case anyone else is having this issue.  It seems like when Python is executed something is attempting to write to /tmp which ends up with a memory error when /tmp is mounted with noexec.

 

Our solution was to add

TMPDIR=<writable path>

to splunk-launch.conf. 

Tags (1)

brettw
Splunk Employee
Splunk Employee

This helped us find a hint of the problem in another environment - thank you for the post!

0 Karma

chris_barrett
SplunkTrust
SplunkTrust

Hi,

We experienced a similar issue when we recently upgraded our TEST environment from 7.3.7 to 8.0.6 - the AppServer and WebUI is working on the 3 clustered Search Heads but isn't working on any of the other servers (eg the Deployment Server).

We've discovered that one difference between the Search Heads and the other servers is that /tmp is mounted with the "noexec" flag on the "other" servers but is not set on the Search Heads.  When I remounted /tmp on the Deployment Server without the noexec flag the AppServer/WebUI started up.

I'm curious to know if your /tmp is set noexec and, if so, if remounting it without the noexec flag "fixes" the issue.

panulpet
Loves-to-Learn

Has memory requirements changed from Splunk version 7.3.3 in the case of Master Node? Now  when running 7.3.3:

MemTotal: 3880788 kB
MemFree: 995200 kB
MemAvailable: 3000696 kB

 

0 Karma

panulpet
Loves-to-Learn

Hi,

 

If there is some who could help I upload splunkd.log with debug logs.. Feel free to check it out 🙂

 

https://www.dropbox.com/s/5w7hnfx65bk3f4k/splunkd.log.gz?dl=0

0 Karma

panulpet
Loves-to-Learn

Now I have tried to upgrade Splunk 4-5 times and everytime same thing...  I have tried "Splunk Platform Upgrade Readiness App" but as I do not have any special apps in Master node that doesn't help anything..

I have tried to start splunk using --debug flag (splunk start --debug") but those millions of lines don't tell me where the problem is.

For me, this seems to some kind of bug in Splunk. The upgrade seems to be so simple that it's difficult to make a mistake there.

I have no clue what to do next?? 

 

0 Karma

panulpet
Loves-to-Learn

Hi,

 

I have now tried to upgrade 3 times and always the same result. I can't access Web UI. Can I attach logs somewhere?

 

This kind of erros seen in log file:

 

10-05-2020 19:39:59.015 +0300 ERROR UiPythonFallback - Couldn't start appserver process on port 8065: Appserver at http://127.0.0.1:8065 never started up. Set `appServerProcessLogStderr` to "true" under [settings] in web.conf. Restart,
try the operation again, and review splunkd.log for any messages that contain "UiAppServer - From appserver".
10-05-2020 19:39:59.015 +0300 ERROR UiPythonFallback - Couldn't start any appserver processes, UI will probably not function correctly!
10-05-2020 19:39:59.015 +0300 ERROR UiHttpListener - No app server is running, stop initializing http server
10-05-2020 19:40:56.949 +0300 WARN GetRemoteAuthToken - Unable to get auth token from peer: https://10.90.136.65:8089 due to: Connection refused

 

I I grep UiAppServer from splunkd.log:

 

grep UiAppServer splunkd.log
10-05-2020 19:39:59.015 +0300 ERROR UiPythonFallback - Couldn't start appserver process on port 8065: Appserver at http://127.0.0.1:8065 never started up. Set `appServerProcessLogStderr` to "true" under [settings] in web.conf. Restart, try the operation again, and review splunkd.log for any messages that contain "UiAppServer - From appserver".
10-05-2020 19:46:46.324 +0300 INFO UiAppServer - Starting stderr collecting thread
10-05-2020 19:46:46.626 +0300 INFO UiAppServer - From appserver: Traceback (most recent call last):
10-05-2020 19:46:46.626 +0300 INFO UiAppServer - From appserver: File "/opt/splunk/lib/python3.7/site-packages/splunk/appserver/mrsparkle/root.py", line 10, in <module>
10-05-2020 19:46:46.626 +0300 INFO UiAppServer - From appserver: from cherrypy import expose
10-05-2020 19:46:46.626 +0300 INFO UiAppServer - From appserver: File "/opt/splunk/lib/python3.7/site-packages/cherrypy/__init__.py", line 76, in <module>
10-05-2020 19:46:46.626 +0300 INFO UiAppServer - From appserver: from . import _cprequest, _cpserver, _cptree, _cplogging, _cpconfig
10-05-2020 19:46:46.626 +0300 INFO UiAppServer - From appserver: File "/opt/splunk/lib/python3.7/site-packages/cherrypy/_cpserver.py", line 6, in <module>
10-05-2020 19:46:46.626 +0300 INFO UiAppServer - From appserver: from cherrypy.process.servers import ServerAdapter
10-05-2020 19:46:46.626 +0300 INFO UiAppServer - From appserver: File "/opt/splunk/lib/python3.7/site-packages/cherrypy/process/__init__.py", line 13, in <module>
10-05-2020 19:46:46.626 +0300 INFO UiAppServer - From appserver: from .wspbus import bus
10-05-2020 19:46:46.626 +0300 INFO UiAppServer - From appserver: File "/opt/splunk/lib/python3.7/site-packages/cherrypy/process/wspbus.py", line 66, in <module>
10-05-2020 19:46:46.626 +0300 INFO UiAppServer - From appserver: import ctypes
10-05-2020 19:46:46.626 +0300 INFO UiAppServer - From appserver: File "/opt/splunk/lib/python3.7/ctypes/__init__.py", line 551, in <module>
10-05-2020 19:46:46.626 +0300 INFO UiAppServer - From appserver: _reset_cache()
10-05-2020 19:46:46.626 +0300 INFO UiAppServer - From appserver: File "/opt/splunk/lib/python3.7/ctypes/__init__.py", line 273, in _reset_cache
10-05-2020 19:46:46.626 +0300 INFO UiAppServer - From appserver: CFUNCTYPE(c_int)(lambda: None)
10-05-2020 19:46:46.626 +0300 INFO UiAppServer - From appserver: MemoryError

 

 

 

0 Karma

richgalloway
SplunkTrust
SplunkTrust

The logs provided are pretty standard stuff, don't indicate any problems that would prevent Splunk from starting, and don't include the error in the title of this posting.

It's not uncommon for the CLI to report "the web interface does not seem to be available" even though the UI *is* available.

Please revert your web.conf changes and try again.

---
If this reply helps you, Karma would be appreciated.
0 Karma

ewok
Observer

Seeing this same issue doing an instance replacement with 9.0.4 or 9.0.5.  Was there ever a solution?

0 Karma

chris_barrett
SplunkTrust
SplunkTrust

Per my response from October 2020, we found the issue to lie with /tmp being flagged as being "noexec".  When we removed this and remount /tmp, the web service started up as expected.  My recollection is that it had something to do with "libffi".

 

0 Karma

ewok
Observer

Thank you Chris.  Do  you (or anyone else) know if this requirement is documented somewhere?  The implementation has security requirement requiring STIG compliance.  Having /tmp mounted exec (without noexec) is a finding.  So, to go with this it needs to be documented as a vendor dependency.  But I haven't found this in anything from Splunk as a requirement.

0 Karma
Get Updates on the Splunk Community!

Introduction to Splunk Observability Cloud - Building a Resilient Hybrid Cloud

Introduction to Splunk Observability Cloud - Building a Resilient Hybrid Cloud  In today’s fast-paced digital ...

Observability protocols to know about

Observability protocols define the specifications or formats for collecting, encoding, transporting, and ...

Take Your Breath Away with Splunk Risk-Based Alerting (RBA)

WATCH NOW!The Splunk Guide to Risk-Based Alerting is here to empower your SOC like never before. Join Haylee ...