Did an upgrade to 8.0 on my development box (Windows w/Splunk 7,.3.1) and the upgrade went without a hitch (as normal).
Did an upgrade to 8.0 on my production Linux box (w/Splunk 7.3.1) and although the upgrade appears to work, it fails to start a webserver. It doesn't even try.
I have verified web.conf:
startwebserver = 1
httpport = 80
enableSplunkWebSSL = false
It has not changed in a number of years. Yet no web-server.
When I restart all of splunk, it never generates the "waiting on splunk web . . . "
When I attempt a 'splunk restart splunkweb' it gives me a very long wait before it says, "Splunk's web interface has been restarted." But there is no interface.
When I do a 'netstat -tulpn | grep :80' I see ports 8089, 8051, and 8089 only. Not 80 or 8000 (in case it reverted for some reason).
And Ideas on what I missed ?
Most recent restart attempt. . .
Shutting down. Please wait, as this may take a few minutes.
[ OK ]
Stopping splunk helpers...
[ OK ]
splunkd.pid doesn't exist...
Splunk> Australian for grep.
Checking http port : open
Checking mgmt port : open
Checking appserver port [127.0.0.1:8065]: open
Checking kvstore port : open
Checking configuration... Done.
Checking critical directories... Done
Validated: audit _internal _introspection _metrics _telemetry _thefishbucket cimmodactions firedalerts history iarchive jmx main os perfmon plw puppet-enterprise summary testing101 uat unixsummary windows wineventlog
Checking filesystem compatibility... Done
Checking conf files for problems...
Checking default conf files for edits...
Validating installed files against hashes from '/opt/splunk/splunk-8.0.0-1357bef0a7f6-linux-2.6-x8664-manifest'
All installed files intact.
All preliminary checks passed.
Starting splunk server daemon (splunkd)...
Notice. No attempt to start webserver or splunkd.
Saw this in the logs:
ERROR UiPythonFallback - Appserver at http://127.0.0.1:8065 never started up!
10-23-2019 08:32:48.695 -0400 ERROR UiPythonFallback - Appserver running on port 8065 exited unexpectedly: exited with code 1
10-23-2019 08:32:48.695 -0400 ERROR UiHttpListener - An applicaiton server has exited unexpectedly, web UI cannot be used until it is restarted
10-23-2019 08:32:48.695 -0400 INFO UiHttpListener - Shutting down webui
10-23-2019 08:32:48.695 -0400 INFO UiHttpListener - Shutting down webui completed
10-23-2019 08:32:48.696 -0400 WARN UiHttpListener - Web UI now stopped
The problem was in "Splunk App for Unix" . . .
File "/opt/splunk/etc/apps/splunkappfor_nix/appserver/modules/CFHiddenSearch/CFHiddenSearch.py", line 65
except splunk.ResourceNotFound, e:
v 7.3.1 to 8.0.0 [patched locally with datetime.xml fix]
4 node SH cluster with 6 node [2 sites] index cluster
The readiness app seems to stop @ the culprit app and not scan any further!!!!
We removed the *NIX app and the Tenable[Nessus] one and all it all started fine and dandy under version 8.
This is due to side-effect of Splunk 8.0 migrating to python 3.0, but some of your existing apps are not fully python 3 compatible. It is very IMPORTANT that you review the About upgrading to 8.0 READ THIS FIRST doc thoroughly before upgrading to Splunk 8.0:
Note: Please pay special attention to the "Changes that can potentially break Splunk Enterprise installations" section of the doc.
Prior to upgrading to 8.0, please also consider running this Splunk Platform Upgrade Readiness App to ensure that some of your existing apps are ready for python 3.0 migration: https://docs.splunk.com/Documentation/UpgradeReadiness/2.0.0/Use/About