Upgrading from 7.0.1 to 8.0 Splunk web stopped running

[vnguyen46]

Hi,
I copied /etc from a production running 7.0.1 to a dev. server, changed all named to match the new server, then upgraded to 8.0.
Splunk web not running as shown here. The website is not accessible. Hope someone can give me an idea how to fix it. Thank you,

[splunk@spkmsaplv001 bin]$ ./splunk restart
Stopping splunkd...
Shutting down. Please wait, as this may take a few minutes.
....... [ OK ]
Stopping splunk helpers...
[ OK ]
Done.

Splunk> The Notorious B.I.G. D.A.T.A.

Checking prerequisites...
Checking http port [8000]: open
Checking mgmt port [8089]: open
Checking appserver port [127.0.0.1:8065]: open
Checking kvstore port [8191]: open
Checking configuration... Done.
Checking critical directories... Done
Checking indexes...
Validated: _audit _internal _introspection _metrics _telemetry _thefishbucket alexa history main optiv summary
Done

Bypassing local license checks since this instance is configured with a remote license master.

    Checking filesystem compatibility...  Done
    Checking conf files for problems...
            Invalid filename value:  $SPLUNK_HOME/etc/apps/GoogleAppsForSplunk/lookups/domains.csv in stanza [gapps_domains] in /opt/splunk/etc/master-apps/TA-GSuiteForSplunk/default/transforms.conf, line 16: only filenames are allowed (not paths)
            Your indexes and inputs configurations are not internally consistent. For more information, run 'splunk btool check --debug'
    Done
    Checking default conf files for edits...
    Validating installed files against hashes from '/opt/splunk/splunk-8.0.0-1357bef0a7f6-linux-2.6-x86_64-manifest'
    All installed files intact.
    Done

All preliminary checks passed.

Starting splunk server daemon (splunkd)...
Done
[ OK ]

Waiting for web server at http://127.0.0.1:8000 to be available............................................................................................................................................................................................................................................................................................................ Done

If you get stuck, we're here to help.
Look for answers here: http://docs.splunk.com

The Splunk web interface is at http://spkmsaplv001.uth.tmc.edu:8000

[AndySplunks]

As a head's up, you've got your Splunk server name in your post in two places. You may want to edit the question to remove the server name.

[splunk@######### bin]$ ./splunk restart

The Splunk web interface is at http://#############:8000

[vnguyen46]

Thank you for the note. I'm looking for a way to edit. Do you know how?

[vnguyen46]

Seems that's the app I will need. For now, I am topped at v7.3.2 and it's working well.

Thanks,

[atammana_splunk]

Copy everything ivanreis said. Additionally, you can download the Splunk platform upgrade readiness app to scan for components that require attention before moving to Enterprise 8.0. Very likely, one of the apps in your environment is using a CherryPy controller and the developer has not yet updated it to be compatible with Python 3.7.

[vnguyen46]

Excellent information. I tested on 7.3.1.1/7.3.2/8.0 and I will stop at at 7.3.2 (from 7.0.1).

Thank you,

[ivanreis · Accepted solution]

Per Splunk document the web interface can run on issues.
https://docs.splunk.com/Documentation/Splunk/8.0.0/Python3Migration/ChangesEnterprise
"Custom web controllers (such as CherryPy endpoints). Requires script-level compatibility with Python 3.7. Failure to make compatible with Python 3.7 may cause issues starting Splunk Web"

Try to setup at server.conf under [general] stanza this configuration
[general]
python.version = python3

I suggest to:

• reinstall the previous splunk version that are working properly
• upgrade splunk to version 7.3.2
• read carefully this document to make sure your environment is properly setup to upgrade to version 8.0:
• https://docs.splunk.com/Documentation/Splunk/8.0.0/Installation/AboutupgradingREADTHISFIRST
• Only upgrade to version 8.0 after you went through all the pre-migration steps first. I suggest you run this process in a separate environment, like a sandbox and test all the apps compatibility before you move on.

[atammana_splunk]

@ivanreis setting python.version=python3 in the [general] stanza in server.conf will make all apps run python3 by default. If apps are not compatible with Python 3 by then, this will cause the apps to fail.

Bits shipped with Splunk Enterprise, including CherryPy, are already running Python 3 by default; setting python.version=python3 will not avoid the problem. The issue is likely with a specific app that the customer is using that leverages CherryPy endpoints that are written in Python 2 only. (The Upgrade Readiness App can help identify this.)