Switching to Python3 in Splunk 8

[rwellum]

I am testing my custom app, which I have converted to be compatible with python2 and python3, on a Splunk 8.0.1 instance.

To force Splunk into using Python3 the documentation[1] states:

Splunk Enterprise version 8.0 include a global setting, python.version, to specify which Python interpreter to use across an instance. The global setting resides in the server.conf file, located in $SPLUNK_HOME/etc/system/local/. The stanza that controls Python version is [general]. For more information how Splunk Enterprise uses configuration files, see About configuration files.
python.version is set to python2 by default in Splunk Enterprise version 8.0. To specify Python 3, set python.version = python3. These settings specify the interpreter to use if a script-level setting has not been specified, but script-level settings take precedence over the global setting, except if Python 3 is forced.
To force Splunk Enterprise to use only the Python 3 interpreter regardless of script-level setting, set python.version = force_python3. Use this setting if you cannot run Python 2.7 past its EOL date of January 1, 2020, or if your Splunk Enterprise deployment and all Splunk apps and add-ons are fully migrated and ready to run Python 3 only

I copied the default server.conf to the local directory and made the change and then restarted splunk with: '/opt/splunk/bin/splunk restart':

[splunk@rich-splunk-centos ~]$ grep python ./splunk/etc/system/local/server.conf
python.version = python3

Afterwards however Splunk default still appears to be python2:

/opt/splunk/bin/splunk cmd python --version
Python 2.7.16

I've repeated with python = force_python3 as the documentation states with the same results.

[splunk@rich-splunk-centos ~]$ grep python ./splunk/etc/system/local/server.conf
python.version = force_python3

I've repeated as well changing: '/opt/splunk/etc/system/local/server.conf' equivalently (in case this was the true splunk home). Again no change.

I've verified a lot of my functionality by running my commands manually like this:

'/opt/splunk/bin/splunk cmd python3 ' - and this is working.

What am I missing? Wondering if the splunk cmd python --version is misleading - because I'm explicitly calling python2 here...

What am I doing wrong? How should I verify which python version is actually running by default?

[hakon_nor · Accepted solution]

If you look into Splunk/bin directory, you will see python, python2 and python3 executables. Running splunk cmd python uses the python executable, which gives you version 2.7.16. Running splunk cmd python3 --version should show the correct python version.

This does not prove that you run the correct version. You can check what version splunk is running as by checking which configuration is active. Doing a splunk btool server list, piping it and select-string/grep for "python" should show you which version Splunk is running as.

[rwellum]

Thank you - this was very helpful