Splunk 8.0 upgrade has no web server running

[tyronetv]

Did an upgrade to 8.0 on my development box (Windows w/Splunk 7,.3.1) and the upgrade went without a hitch (as normal).
Did an upgrade to 8.0 on my production Linux box (w/Splunk 7.3.1) and although the upgrade appears to work, it fails to start a webserver. It doesn't even try.

I have verified web.conf:
[settings]
startwebserver = 1
httpport = 80
enableSplunkWebSSL = false

It has not changed in a number of years. Yet no web-server.

When I restart all of splunk, it never generates the "waiting on splunk web . . . "

When I attempt a 'splunk restart splunkweb' it gives me a very long wait before it says, "Splunk's web interface has been restarted." But there is no interface.

When I do a 'netstat -tulpn | grep :80' I see ports 8089, 8051, and 8089 only. Not 80 or 8000 (in case it reverted for some reason).

And Ideas on what I missed ?

[jfeitosa_real]

I also had the same error after upgrading to version 8.0.2.1. However, the previous version was 6.3.0 and there was no upgrade to version 7.0.0 and then to 8.0.2.1.
We had to downgrade to 7.0.0, then move up to 8.0.2.1, but the errors continued.

web_service.log:
2020-03-24 17: 06: 04,514 ERROR [5e7a682c3a7fa117129950] root: 769 - Unable to start splunkweb
2020-03-24 17: 06: 04,514 ERROR [5e7a682c3a7fa117129950] root: 770 - invalid syntax (CustomRESTForSavedSearch.py, line 24)
Traceback (most recent call last):
  File "/opt/splunk/lib/python3.7/site-packages/splunk/appserver/mrsparkle/root.py", line 132, in
    from splunk.appserver.mrsparkle.controllers.top import TopController
  File "/opt/splunk/lib/python3.7/site-packages/splunk/appserver/mrsparkle/controllers/top.py", line 27, in
    from splunk.appserver.mrsparkle.controllers.admin import AdminController
  File "/opt/splunk/lib/python3.7/site-packages/splunk/appserver/mrsparkle/controllers/admin.py", line 25, in
    from splunk.appserver.mrsparkle.controllers.appinstall import AppInstallController
  File "/opt/splunk/lib/python3.7/site-packages/splunk/appserver/mrsparkle/controllers/appinstall.py", line 22, in
    from splunk.appserver.mrsparkle.lib import module
  File "/opt/splunk/lib/python3.7/site-packages/splunk/appserver/mrsparkle/lib/module.py", line 465, in
    moduleMapper = ModuleMapper ()
  File "/opt/splunk/lib/python3.7/site-packages/splunk/appserver/mrsparkle/lib/module.py", line 83, in init
    self.installedModules = self.getInstalledModules ()
  File "/opt/splunk/lib/python3.7/site-packages/splunk/appserver/mrsparkle/lib/module.py", line 28, in helper
    return f (* a, ** kw)
  File "/opt/splunk/lib/python3.7/site-packages/splunk/appserver/mrsparkle/lib/module.py", line 448, in getInstalledModules
    mods = self.getModuleList (root)
  File "/opt/splunk/lib/python3.7/site-packages/splunk/appserver/mrsparkle/lib/module.py", line 37, in helper
    return f (* a, ** kw)
  File "/opt/splunk/lib/python3.7/site-packages/splunk/appserver/mrsparkle/lib/module.py", line 223, in getModuleList
    mod = _import _ (modname)
  File "/opt/splunk/etc/apps/sideview_utils/appserver/modules/CustomRESTForSavedSearch/CustomRESTForSavedSearch.py", line 24
    except Exception, and:
                    ^
SyntaxError: invalid syntax

- splunkd.log:
03-24-2020 17: 35: 05.961 -0300 ERROR TailReader - File will not be read, is too small to match seekptr checksum (file = / opt / splunk / var / log / splunk / migration.log.2020-03- 24.17-05-51). Last time we saw this initcrc, filename was different. You may wish to use larger initCrcLen for this sourcetype, or a CRC salt on this source. Consult the documentation or file a support case online at http://www.splunk.com/page/submit_issue for more info.
03-24-2020 17: 35: 06.329 -0300 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/splunk_deployment_monitor/bin/scripted_inputs/ftr_upgrade.py" Traceback (most recent call last):
03-24-2020 17: 35: 06.329 -0300 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/splunk_deployment_monitor/bin/scripted_inputs/ftr_upgrade.py" File "/ opt / splunk / etc / apps / splunk_deployment_monitor /bin/scripted_inputs/ftr_upgrade.py ", line 27, in
03-24-2020 17: 35: 06.330 -0300 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/splunk_deployment_monitor/bin/scripted_inputs/ftr_upgrade.py" build = int (server ['build'])
03-24-2020 17: 35: 06.330 -0300 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/splunk_deployment_monitor/bin/scripted_inputs/ftr_upgrade.py" ValueError: invalid literal for int () with base 10: 'c8a78efdd40f'
03-24-2020 17: 35: 13.378 -0300 ERROR regexExtractionProcessor - REGEX field must be specified tranform_name = introspection_resource_usage_metric_name
03-24-2020 17: 47: 50.081 -0300 ERROR TcpInputProc - Error encountered for connection from src = 10.100.4.44: 54677. Local side shutting down
03-24-2020 17: 52: 02.502 -0300 ERROR LMStack - License should have extension of '.lic' or '.license', ignoring file 383994_82166_Autral_Reset_Splunk_030220.License
03-24-2020 17: 52: 02.954 -0300 ERROR MongodRunner - mongod exited abnormally (exit code 14, status: exited with code 14) - look at mongod.log to investigate.
03-24-2020 17: 52: 02.955 -0300 ERROR KVStoreBulletinBoardManager - KV Store process terminated abnormally (exit code 14, status exited with code 14). See mongod.log and splunkd.log for details.
03-24-2020 17: 52: 02.955 -0300 ERROR KVStoreBulletinBoardManager - KV Store changed status to failed. KVStore process terminated.
03-24-2020 17: 52: 03.062 -0300 ERROR TcpInputProc - Could not bind to port IPv4 port 514
03-24-2020 17: 52: 03.062 -0300 ERROR TcpInputProc - Could not bind to port IPv4 port 520
03-24-2020 17: 52: 03.155 -0300 ERROR UDPInputProcessor - Error binding to socket in UDPInputProcessor: Permission denied
03-24-2020 17: 52: 03.156 -0300 ERROR FrameworkUtils

What can be done now?

Thank in advanced!

[woodcock]

Come back here @tyronetv and click Accept to close this question.

[woodcock]

Check your web_service.log and at the very bottom it will identify the app that is causing your problem, something like this:

File "C:\Program Files\Splunk\etc\apps\<app_that_is_causing_the_problem_here>\appserver\modules\CustomRESTForSavedSearch\CustomRESTForSavedSearch.py", line 24
     except Exception, e:
                     ^
 SyntaxError: invalid syntax

The quick solution is to go to the CLI on your Search head, and create a $SPLUNK_HOME/etc/apps/<app_that_is_causing_the_problem_here>/local/app.conf file with the contents below to disable it temporarily and then restart. Splunk will come up and you can then upgrade to the latest version of app_that_is_causing_the_problem_here, live without it, or work with the author to fix the problem if it still exists on the latest version:

[install]
state = disabled

To avoid some of these kinds of problems in the future, be sure to run the Splunk Platform Upgrade Readiness App app before upgrading:
https://splunkbase.splunk.com/app/4698/

[splunkIT]

This is due to side-effect of Splunk 8.0 migrating to python 3.0, but some of your existing apps are not fully python 3 compatible. It is very IMPORTANT that you review the About upgrading to 8.0 READ THIS FIRST doc thoroughly before upgrading to Splunk 8.0:

https://docs.splunk.com/Documentation/Splunk/8.0.0/Installation/AboutupgradingREADTHISFIRST
Note: Please pay special attention to the "Changes that can potentially break Splunk Enterprise installations" section of the doc.

Prior to upgrading to 8.0, please also consider running this Splunk Platform Upgrade Readiness App to ensure that some of your existing apps are ready for python 3.0 migration: https://docs.splunk.com/Documentation/UpgradeReadiness/2.0.0/Use/About

[wgawhh5hbnht]

If you did upgrade to Splunk 8.0 & didn't run the Readiness App and the WebUI isn't running, try this:

Check the /opt/splunk/var/log/splunk/web_service.log
You're looking for something like this:

2019-11-26 23:29:03,566 ERROR [5dddb53f357fd0e7c7cf90] root:769 - Unable to start splunkweb
2019-11-26 23:29:03,566 ERROR [5dddb53f357fd0e7c7cf90] root:770 - 'dict' object has no attribute 'iteritems'
Traceback (most recent call last):
File "/opt/splunk/lib/python3.7/site-packages/splunk/appserver/mrsparkle/root.py", line 132, in <module>
from splunk.appserver.mrsparkle.controllers.top import TopController
File "/opt/splunk/lib/python3.7/site-packages/splunk/appserver/mrsparkle/controllers/top.py", line 27, in <module>
from splunk.appserver.mrsparkle.controllers.admin import AdminController
File "/opt/splunk/lib/python3.7/site-packages/splunk/appserver/mrsparkle/controllers/admin.py", line 17, in <module>
import formencode
File "/opt/splunk/lib/python3.7/site-packages/formencode/__init__.py", line 9, in <module>
from formencode import validators
File "/opt/splunk/lib/python3.7/site-packages/formencode/validators.py", line 15, in <module>
import dns.resolver
File "/opt/splunk/etc/apps/generateblocklist_app/bin/dns/resolver.py", line 32, in <module>
import dns.flags
File "/opt/splunk/etc/apps/generateblocklist_app/bin/dns/flags.py", line 51, in <module>
_by_value = dict([(y, x) for x, y in _by_text.iteritems()])
AttributeError: 'dict' object has no attribute 'iteritems'

Where there is an attribute error.
Move the offending app to disabled and attempt to restart splunk again. You may have to do this several times depending on if there are multiple offending apps or not.
Deleting generateblocklist app solved the splunk webui not starting for me.

[tyronetv]

Found it.

The problem was in "Splunk App for Unix" . . .

File "/opt/splunk/etc/apps/splunk_app_for_nix/appserver/modules/CFHiddenSearch/CFHiddenSearch.py", line 65
except splunk.ResourceNotFound, e:

[woodcock]

Come back here @tyronetv and click Accept to close this question.

[deput_d]

cd $SPLUNK_HOME
mv etc/apps/splunk_app_for_nix etc/disabled-apps/
mv etc/apps/Splunk_TA_nix etc/disabled-apps/

[barriejames]

App not on our setup but having same issue.

[smithy001]

v 7.3.1 to 8.0.0 [patched locally with datetime.xml fix]

4 node SH cluster with 6 node [2 sites] index cluster

The readiness app seems to stop @ the culprit app and not scan any further!!!!

We removed the *NIX app and the Tenable[Nessus] one and all it all started fine and dandy under version 8.

[benstanding]

I moved this App into the disabled-apps and the Web UI opened just fine.

[tyronetv]

Saw this in the logs:

ERROR UiPythonFallback - Appserver at http://127.0.0.1:8065 never started up!

10-23-2019 08:32:48.695 -0400 ERROR UiPythonFallback - Appserver running on port 8065 exited unexpectedly: exited with code 1
10-23-2019 08:32:48.695 -0400 ERROR UiHttpListener - An applicaiton server has exited unexpectedly, web UI cannot be used until it is restarted
10-23-2019 08:32:48.695 -0400 INFO UiHttpListener - Shutting down webui
10-23-2019 08:32:48.695 -0400 INFO UiHttpListener - Shutting down webui completed
10-23-2019 08:32:48.696 -0400 WARN UiHttpListener - Web UI now stopped

[tyronetv]

Most recent restart attempt. . .

==============

splunk restart

Stopping splunkd...
Shutting down. Please wait, as this may take a few minutes.
[ OK ]
Stopping splunk helpers...
[ OK ]
Done.
splunkd.pid doesn't exist...

Splunk> Australian for grep.

Checking prerequisites...
Checking http port [80]: open
Checking mgmt port [8089]: open
Checking appserver port [127.0.0.1:8065]: open
Checking kvstore port [8191]: open
Checking configuration... Done.
Checking critical directories... Done
Checking indexes...
Validated: _audit _internal _introspection _metrics _telemetry _thefishbucket cim_modactions firedalerts history iarchive jmx main os perfmon plw puppet-enterprise summary testing101 uat unix_summary windows wineventlog
Done
Checking filesystem compatibility... Done
Checking conf files for problems...

Done
Checking default conf files for edits...
Validating installed files against hashes from '/opt/splunk/splunk-8.0.0-1357bef0a7f6-linux-2.6-x86_64-manifest'
All installed files intact.
Done
All preliminary checks passed.

Starting splunk server daemon (splunkd)...
Done

[ OK ]

Notice. No attempt to start webserver or splunkd.