Archive
Highlighted

Splunk 8.0 upgrade has no web server running

Communicator

Did an upgrade to 8.0 on my development box (Windows w/Splunk 7,.3.1) and the upgrade went without a hitch (as normal).
Did an upgrade to 8.0 on my production Linux box (w/Splunk 7.3.1) and although the upgrade appears to work, it fails to start a webserver. It doesn't even try.

I have verified web.conf:
[settings]
startwebserver = 1
httpport = 80
enableSplunkWebSSL = false

It has not changed in a number of years. Yet no web-server.

When I restart all of splunk, it never generates the "waiting on splunk web . . . "

When I attempt a 'splunk restart splunkweb' it gives me a very long wait before it says, "Splunk's web interface has been restarted." But there is no interface.

When I do a 'netstat -tulpn | grep :80' I see ports 8089, 8051, and 8089 only. Not 80 or 8000 (in case it reverted for some reason).

And Ideas on what I missed ?

0 Karma
Highlighted

Re: Splunk 8.0 upgrade has no web server running

Communicator

Most recent restart attempt. . .

==============

splunk restart

Stopping splunkd...
Shutting down. Please wait, as this may take a few minutes.
[ OK ]
Stopping splunk helpers...
[ OK ]
Done.
splunkd.pid doesn't exist...

Splunk> Australian for grep.

Checking prerequisites...
Checking http port [80]: open
Checking mgmt port [8089]: open
Checking appserver port [127.0.0.1:8065]: open
Checking kvstore port [8191]: open
Checking configuration... Done.
Checking critical directories... Done
Checking indexes...
Validated: audit _internal _introspection _metrics _telemetry _thefishbucket cimmodactions firedalerts history iarchive jmx main os perfmon plw puppet-enterprise summary testing101 uat unixsummary windows wineventlog
Done
Checking filesystem compatibility... Done
Checking conf files for problems...

Done
Checking default conf files for edits...
Validating installed files against hashes from '/opt/splunk/splunk-8.0.0-1357bef0a7f6-linux-2.6-x86
64-manifest'
All installed files intact.
Done
All preliminary checks passed.

Starting splunk server daemon (splunkd)...
Done

[ OK ]

Notice. No attempt to start webserver or splunkd.

0 Karma
Highlighted

Re: Splunk 8.0 upgrade has no web server running

Communicator

Saw this in the logs:

ERROR UiPythonFallback - Appserver at http://127.0.0.1:8065 never started up!

10-23-2019 08:32:48.695 -0400 ERROR UiPythonFallback - Appserver running on port 8065 exited unexpectedly: exited with code 1
10-23-2019 08:32:48.695 -0400 ERROR UiHttpListener - An applicaiton server has exited unexpectedly, web UI cannot be used until it is restarted
10-23-2019 08:32:48.695 -0400 INFO UiHttpListener - Shutting down webui
10-23-2019 08:32:48.695 -0400 INFO UiHttpListener - Shutting down webui completed
10-23-2019 08:32:48.696 -0400 WARN UiHttpListener - Web UI now stopped

0 Karma
Highlighted

Re: Splunk 8.0 upgrade has no web server running

Communicator

Found it.

The problem was in "Splunk App for Unix" . . .

File "/opt/splunk/etc/apps/splunkappfor_nix/appserver/modules/CFHiddenSearch/CFHiddenSearch.py", line 65
except splunk.ResourceNotFound, e:

Highlighted

Re: Splunk 8.0 upgrade has no web server running

Explorer

App not on our setup but having same issue.

0 Karma
Highlighted

Re: Splunk 8.0 upgrade has no web server running

Engager

I moved this App into the disabled-apps and the Web UI opened just fine.

0 Karma
Highlighted

Re: Splunk 8.0 upgrade has no web server running

New Member

v 7.3.1 to 8.0.0 [patched locally with datetime.xml fix]

4 node SH cluster with 6 node [2 sites] index cluster

The readiness app seems to stop @ the culprit app and not scan any further!!!!

We removed the *NIX app and the Tenable[Nessus] one and all it all started fine and dandy under version 8.

0 Karma
Highlighted

Re: Splunk 8.0 upgrade has no web server running

Engager

cd $SPLUNKHOME
mv etc/apps/splunk
appfornix etc/disabled-apps/
mv etc/apps/SplunkTAnix etc/disabled-apps/

0 Karma
Highlighted

Re: Splunk 8.0 upgrade has no web server running

Esteemed Legend

Come back here @tyronetv and click Accept to close this question.

0 Karma
Highlighted

Re: Splunk 8.0 upgrade has no web server running

Splunk Employee
Splunk Employee

This is due to side-effect of Splunk 8.0 migrating to python 3.0, but some of your existing apps are not fully python 3 compatible. It is very IMPORTANT that you review the About upgrading to 8.0 READ THIS FIRST doc thoroughly before upgrading to Splunk 8.0:

https://docs.splunk.com/Documentation/Splunk/8.0.0/Installation/AboutupgradingREADTHISFIRST
Note: Please pay special attention to the "Changes that can potentially break Splunk Enterprise installations" section of the doc.

Prior to upgrading to 8.0, please also consider running this Splunk Platform Upgrade Readiness App to ensure that some of your existing apps are ready for python 3.0 migration: https://docs.splunk.com/Documentation/UpgradeReadiness/2.0.0/Use/About

0 Karma