Universal Forwarder 8.x missing python2/3 on non enterprise servers

[Kieren]

Hi Splunk Gurus

Im hoping that there is a simple answer for this issue.

We have recently upgraded to Splunk Enterprise 8.2.
Our servers (RHEL 7/8) are all running Universal Forwarders 8.0.

The issue we have found is that the UF does not include the Python 2.7/3.7 binaries and libs as part of its install package (yes I know this has not been the case for a long time).
This is not an issue if you are installing the forwarder on a Splunk Node as the Enterprise version includes these and installs them (as far as I can tell) into the correct locations in the forwarder for it to use internally.

The problem appears when trying to upgrade the standalone linux package (.tgz or .rpm) to 8.2.2.1 as the binary and packages for python3.7 are required (regardless of python.version setting)  to run the migration upgrade scripts

As RHEL7/8 only has a supported package for Python 3.6 this becomes an even more pressing issue.
I have installed Python 3.7 from source to try as a workaround and linked it to /opt/splunkforwarder/bin/python3.7 with some success.

The main problem seems to be that the site-packages path seems to be hard coded into the forwarder to look for packages in the /opt/splunkforwarder/lib/python3.7/site-packages
regardless of the python lib path locations.

eg if I symlink /usr/local/bin/python3.7  -> /opt/splunk/forwarder/bin/python3.7
I get these kinds of errors in the splunkd.log
/opt/splunkforwarder/bin/python3.7: can't open file '/opt/splunkforwarder/lib/python3.7/site-packages/splunk/clilib/cli.py': [Errno 2] No such file or directory
As the splunk cmd which runs python scripts from apps cannot even start correctly regardless of the python.version value set in the app or server.conf

So my actual question is how do we get the python 2.7 & 3.7 binaries and associated required packages into a forwarder?
Is there a .tgz or .rpm that we can use to get the internal python versions the forwarder requires installed in the right locations?
Or a full forwarder .rpm that includes the binaries for exactly this standalone purpose?

This would seem to be a significant oversight that assumes Splunk Enterpise will always be available to use as a base installer for all servers, and additionally that python 3.7 is always available/easily installed.

A much less desirable option would be to roll back the forwarders (and all deployed apps to the latest 7.x version) but this limits moving forward and will vreate many more compatibility issues than it will solve

Any helpful hints pointers or advice would be greatly appreciated

Regards

Kieren

[PickleRick]

@Kierenwhy would you want to install an UF on a splunk server???

@wduckettThere's nothing to be solved here. UF is not supposed to include python. It's not a bug, it's a feature. If you need python, install Heavy Forwarder or use external python installation (but if I'm not mistaken there can still be some issues with running python-based modular inputs in this case).

[VatsalJagani]

@Kieren , @wduckett  - UF does not suppose to include python with the bundle. You can use Python externally if you want.

I hope this helps!!!

[wduckett]

Ever figure this out? Having the same issue...