Unable to access Splunk Web receiving error message

[jegger]

VM = Windows 2012
Splunk = v8.0

We have not been able to access the Web UI. Within the web_service.log, there is an error which states the following:

2019-11-22 07:59:23,709 ERROR   [5dd805ceba3b897dc898] root:709 - Unable to start splunkweb
2019-11-22 07:59:23,710 ERROR   [5dd805ceba3b897dc898] root:710 - ("no 'pooling' stanza exists in server.conf.  Your configuration may be corrupt or may require a restart.",)
Traceback (most recent call last):
  File "C:\Program Files\Splunk\Python-2.7\Lib\site-packages\splunk\appserver\mrsparkle\root.py", line 705, in 
    run(blocking=True)
  File "C:\Program Files\Splunk\Python-2.7\Lib\site-packages\splunk\appserver\mrsparkle\root.py", line 238, in run
    pooling_cfg = splunk_to_cherry_cfg('server', 'pooling')
  File "C:\Program Files\Splunk\Python-2.7\Lib\site-packages\splunk\appserver\mrsparkle\lib\util.py", line 504, in splunk_to_cherry_cfg
    for k,v in splunk.clilib.cli_common.getConfStanza(conf, stanza).items():
  File "C:\Program Files\Splunk\Python-2.7\Lib\site-packages\splunk\clilib\cli_common.py", line 357, in getConfStanza
    raise ParsingError("no '%s' stanza exists in %s.conf.  Your configuration may be corrupt or may require a restart." % (stanza, confName))
ParsingError: ("no 'pooling' stanza exists in server.conf.  Your configuration may be corrupt or may require a restart.",)
2019-11-22 08:21:39,425 INFO    [5dd80b118caa9c80ed88] __init__:174 - Using default logging config file: C:\Program Files\Splunk\etc\log.cfg
2019-11-22 08:21:39,425 INFO    [5dd80b118caa9c80ed88] __init__:212 - Setting logger=splunk level=INFO
2019-11-22 08:21:39,425 INFO    [5dd80b118caa9c80ed88] __init__:212 - Setting logger=splunk.appserver level=INFO
2019-11-22 08:21:39,425 INFO    [5dd80b118caa9c80ed88] __init__:212 - Setting logger=splunk.appserver.controllers level=INFO
2019-11-22 08:21:39,425 INFO    [5dd80b118caa9c80ed88] __init__:212 - Setting logger=splunk.appserver.controllers.proxy level=INFO
2019-11-22 08:21:39,425 INFO    [5dd80b118caa9c80ed88] __init__:212 - Setting logger=splunk.appserver.lib level=WARN
2019-11-22 08:21:39,425 INFO    [5dd80b118caa9c80ed88] __init__:212 - Setting logger=splunk.pdfgen level=INFO
2019-11-22 08:21:39,425 INFO    [5dd80b118caa9c80ed88] __init__:212 - Setting logger=splunk.archiver_restoration level=INFO
2019-11-22 08:21:40,425 ERROR   [5dd80b118caa9c80ed88] root:769 - Unable to start splunkweb
2019-11-22 08:21:40,425 ERROR   [5dd80b118caa9c80ed88] root:770 - invalid syntax (CustomRESTForSavedSearch.py, line 24)
Traceback (most recent call last):
  File "C:\Program Files\Splunk\Python-3.7\Lib\site-packages\splunk\appserver\mrsparkle\root.py", line 132, in 
    from splunk.appserver.mrsparkle.controllers.top import TopController
  File "C:\Program Files\Splunk\Python-3.7\lib\site-packages\splunk\appserver\mrsparkle\controllers\top.py", line 27, in 
    from splunk.appserver.mrsparkle.controllers.admin import AdminController
  File "C:\Program Files\Splunk\Python-3.7\lib\site-packages\splunk\appserver\mrsparkle\controllers\admin.py", line 25, in 
    from splunk.appserver.mrsparkle.controllers.appinstall import AppInstallController
  File "C:\Program Files\Splunk\Python-3.7\lib\site-packages\splunk\appserver\mrsparkle\controllers\appinstall.py", line 22, in 
    from splunk.appserver.mrsparkle.lib import module
  File "C:\Program Files\Splunk\Python-3.7\lib\site-packages\splunk\appserver\mrsparkle\lib\module.py", line 465, in 
    moduleMapper = ModuleMapper()
  File "C:\Program Files\Splunk\Python-3.7\lib\site-packages\splunk\appserver\mrsparkle\lib\module.py", line 83, in __init__
    self.installedModules = self.getInstalledModules()
  File "C:\Program Files\Splunk\Python-3.7\lib\site-packages\splunk\appserver\mrsparkle\lib\module.py", line 28, in helper
    return f(*a, **kw)
  File "C:\Program Files\Splunk\Python-3.7\lib\site-packages\splunk\appserver\mrsparkle\lib\module.py", line 448, in getInstalledModules
    mods = self.getModuleList(root)
  File "C:\Program Files\Splunk\Python-3.7\lib\site-packages\splunk\appserver\mrsparkle\lib\module.py", line 37, in helper
    return f(*a, **kw)
  File "C:\Program Files\Splunk\Python-3.7\lib\site-packages\splunk\appserver\mrsparkle\lib\module.py", line 223, in getModuleList
    mod = __import__(modname)
  File "C:\Program Files\Splunk\etc\apps\sideview_utils\appserver\modules\CustomRESTForSavedSearch\CustomRESTForSavedSearch.py", line 24
    except Exception, e:
                    ^
SyntaxError: invalid syntax

How can we resolve this issue?

Let me know if there are any other logs I should provide.

[woodcock]

The quick solution is to go to the CLI on your Search head, and create a $SPLUNK_HOME/etc/apps/sideview_utils/local/app.conf file with the contents below to disable it temporarily and then restart. Splunk will come up and you can then upgrade to the latest version of sideview_utils, live without it, or work with @sideview to fix the problem if it still exists on the latest version (HIGHLY unlikely):

[install]
state = disabled

To avoid some of these kinds of problems in the future, be sure to run the Splunk Platform Upgrade Readiness App app before upgrading:
https://splunkbase.splunk.com/app/4698/

[uagrawal_splunk]

From logs, it seems like the pooling stanza is missing in the default/server.conf file. Add the pooling stanza https://docs.splunk.com/Documentation/Splunk/8.0.0/Admin/Serverconf and the web UI will be accessible.

[jegger]

Thank you. Running a bit blind over here due to lack of Splunk knowledge. The admin who configured it is no longer with the company.

I added the below pooling stanza, but still cannot access web:
[pooling]
state = disabled

Rebooted the server and don't see a newer timestamped pooling error, but I'm still seeing the following:

2019-11-22 10:22:29,887 ERROR [5dd82763dbda51fa17c8] root:769 - Unable to start splunkweb
2019-11-22 10:22:29,887 ERROR [5dd82763dbda51fa17c8] root:770 - invalid syntax (CustomRESTForSavedSearch.py, line 24)
Traceback (most recent call last):
File "C:\Program Files\Splunk\Python-3.7\Lib\site-packages\splunk\appserver\mrsparkle\root.py", line 132, in
from splunk.appserver.mrsparkle.controllers.top import TopController
File "C:\Program Files\Splunk\Python-3.7\lib\site-packages\splunk\appserver\mrsparkle\controllers\top.py", line 27, in
from splunk.appserver.mrsparkle.controllers.admin import AdminController
File "C:\Program Files\Splunk\Python-3.7\lib\site-packages\splunk\appserver\mrsparkle\controllers\admin.py", line 25, in
from splunk.appserver.mrsparkle.controllers.appinstall import AppInstallController
File "C:\Program Files\Splunk\Python-3.7\lib\site-packages\splunk\appserver\mrsparkle\controllers\appinstall.py", line 22, in
from splunk.appserver.mrsparkle.lib import module
File "C:\Program Files\Splunk\Python-3.7\lib\site-packages\splunk\appserver\mrsparkle\lib\module.py", line 465, in
moduleMapper = ModuleMapper()
File "C:\Program Files\Splunk\Python-3.7\lib\site-packages\splunk\appserver\mrsparkle\lib\module.py", line 83, in init
self.installedModules = self.getInstalledModules()
File "C:\Program Files\Splunk\Python-3.7\lib\site-packages\splunk\appserver\mrsparkle\lib\module.py", line 28, in helper
return f(*a, **kw)
File "C:\Program Files\Splunk\Python-3.7\lib\site-packages\splunk\appserver\mrsparkle\lib\module.py", line 448, in getInstalledModules
mods = self.getModuleList(root)
File "C:\Program Files\Splunk\Python-3.7\lib\site-packages\splunk\appserver\mrsparkle\lib\module.py", line 37, in helper
return f(*a, **kw)
File "C:\Program Files\Splunk\Python-3.7\lib\site-packages\splunk\appserver\mrsparkle\lib\module.py", line 223, in getModuleList
mod = import(modname)
File "C:\Program Files\Splunk\etc\apps\sideview_utils\appserver\modules\CustomRESTForSavedSearch\CustomRESTForSavedSearch.py", line 24
except Exception, e:
^
SyntaxError: invalid syntax
2019-11-22 10:31:51,890 INFO [5dd8299447b81562ecc8] init:174 - Using default logging config file: C:\Program Files\Splunk\etc\log.cfg
2019-11-22 10:31:51,937 INFO [5dd8299447b81562ecc8] init:212 - Setting logger=splunk level=INFO
2019-11-22 10:31:51,937 INFO [5dd8299447b81562ecc8] init:212 - Setting logger=splunk.appserver level=INFO
2019-11-22 10:31:51,937 INFO [5dd8299447b81562ecc8] init:212 - Setting logger=splunk.appserver.controllers level=INFO
2019-11-22 10:31:51,937 INFO [5dd8299447b81562ecc8] init:212 - Setting logger=splunk.appserver.controllers.proxy level=INFO
2019-11-22 10:31:51,937 INFO [5dd8299447b81562ecc8] init:212 - Setting logger=splunk.appserver.lib level=WARN
2019-11-22 10:31:51,937 INFO [5dd8299447b81562ecc8] init:212 - Setting logger=splunk.pdfgen level=INFO
2019-11-22 10:31:51,937 INFO [5dd8299447b81562ecc8] init:212 - Setting logger=splunk.archiver_restoration level=INFO
2019-11-22 10:31:54,093 ERROR [5dd8299447b81562ecc8] root:769 - Unable to start splunkweb
2019-11-22 10:31:54,093 ERROR [5dd8299447b81562ecc8] root:770 - invalid syntax (CustomRESTForSavedSearch.py, line 24)
Traceback (most recent call last):
File "C:\Program Files\Splunk\Python-3.7\Lib\site-packages\splunk\appserver\mrsparkle\root.py", line 132, in
from splunk.appserver.mrsparkle.controllers.top import TopController
File "C:\Program Files\Splunk\Python-3.7\lib\site-packages\splunk\appserver\mrsparkle\controllers\top.py", line 27, in
from splunk.appserver.mrsparkle.controllers.admin import AdminController
File "C:\Program Files\Splunk\Python-3.7\lib\site-packages\splunk\appserver\mrsparkle\controllers\admin.py", line 25, in
from splunk.appserver.mrsparkle.controllers.appinstall import AppInstallController
File "C:\Program Files\Splunk\Python-3.7\lib\site-packages\splunk\appserver\mrsparkle\controllers\appinstall.py", line 22, in
from splunk.appserver.mrsparkle.lib import module
File "C:\Program Files\Splunk\Python-3.7\lib\site-packages\splunk\appserver\mrsparkle\lib\module.py", line 465, in
moduleMapper = ModuleMapper()
File "C:\Program Files\Splunk\Python-3.7\lib\site-packages\splunk\appserver\mrsparkle\lib\module.py", line 83, in init
self.installedModules = self.getInstalledModules()
File "C:\Program Files\Splunk\Python-3.7\lib\site-packages\splunk\appserver\mrsparkle\lib\module.py", line 28, in helper
return f(*a, **kw)
File "C:\Program Files\Splunk\Python-3.7\lib\site-packages\splunk\appserver\mrsparkle\lib\module.py", line 448, in getInstalledModules
mods = self.getModuleList(root)
File "C:\Program Files\Splunk\Python-3.7\lib\site-packages\splunk\appserver\mrsparkle\lib\module.py", line 37, in helper
return f(*a, **kw)
File "C:\Program Files\Splunk\Python-3.7\lib\site-packages\splunk\appserver\mrsparkle\lib\module.py", line 223, in getModuleList
mod = import(modname)
File "C:\Program Files\Splunk\etc\apps\sideview_utils\appserver\modules\CustomRESTForSavedSearch\CustomRESTForSavedSearch.py", line 24
except Exception, e:
^
SyntaxError: invalid syntax