Cannot use pip from pyden

[mmccul_fe]

Installed pyden 1.2 and pyden-manager, restarted Splunk, but get errors every time I use the | pip command.

I installed two versions of python, python 3.7.3 and 2.7.16 using the | createdist and | createvenv commands. Validated by running a | getvenvs command. Output as below:

is_default  version environment
1   2.7.16  abc123
0   3.7.3   def456

I try to install a simple package by | pip install panda and get

"External search command 'pip' returned error code 1. ."

No hints on what the cause of the error is at all.

When I look at the underlying filestructure, I don't see the file structure I would expect under venv based on the pyden.conf file: $SPLUNK_HOME/etc/apps/pyden/local/lib/venv is empty.

Any suggestions?

I also cannot | pydelete anything to destroy the venv and start over.

[badarsebard · Accepted solution]

Update 2

Ok, I've got everything worked out. So I've found all the environment variables I need to unset so that each version of Python can build using only host packages/libraries and not using things built into Splunk. I've also found that for this specific case (Ubuntu 14), the OpenSSL package in the standard repository is of insufficient version for Python 3.7 (required: 1.0.2, highest available: 1.0.1). So in order to resolve OP's specific issue three steps will be needed:

1. Update PyDen to the latest release (1.2.1)
2. Ensure the following packages are installed on the ubuntu box:
   • build-essential
   • libffi-dev
   • libssl-dev
3. Use Python version 3.6 or lower.

Once completed, I've verified I can create new virtual environments and successfully install pandas to them. Let me know if you have any further issues and please accept if this solution worked.

---

Update

After some more digging I've discovered two issues at play. The first is what I discussed below is that the cause of failing to create new environments stems from pip being unable to use SSL because the correct OpenSSL library is unavailable on the host. There's a second issue at play when it comes to the libraries as well. When Splunk runs a Python script it creates a set of environment variables that point to some libraries bundled with the core software, one of which is its own OpenSSL. So even if you have the correct packages installed on the host they won't necessarily be used during the make process because of the Splunk environment variables. I'm working on a hotfix to remove those library references so the make process will default to the host shared libraries. Should be forth coming in a day or two.

---

So after digging in to the issue (after getting the OS version from OP) it looks like the issue is most likely one of dependencies for Python compilation. Since PyDen downloads the Python source code and compiles it on the host directly, the host has to have the correct set of packages installed in order for the build to be successful. The behavior described by OP in slack conversation indicates that, while Python is compiling successfully, it isn't bundling pip properly. This is evidenced by the fact that the Python distribution gets created successfully but virtual environments fail to build.

CPython compilation is built to be as fault tolerant as possible, so Python can compile successfully with a number of core modules missing. Which of these modules will be unavailable is dependent on the host libraries installed. For Ubuntu, you typically need at least build-essential, libffi-dev, and libssl-dev. But again, this varies by the exact version and I haven't yet found the minimum correct set of packages needed for the version.

I'll keep working on finding the minimal set of packages, but in the mean time I'd suggest reviewing the script I use to provision my Vagrant boxes for development here. Try the packages I'm installing with APT and see if you're able to create new versions that can build virtual environments successfully. I use the virtualenv package to create the environments and it's installed through pip, so if you're able to build the environment you should be able to add packages through pip.

I'll keep looking for a set of packages I know will work and will update when I find it.

[spammenot66]

@badarsebard  same here. I've installed both pyden and pyden-manager. It throws an error for me when i try to run the createvenv and pip command. Do i need to install it in a specific order - pyden then pyden-manager, or vice versa?

When i run

| createvenv version=3.7.2 name=mypy

I get the following error: External search command 'createvenv' returned error code 1. . 

| pip install requests

I get the following error: External search command 'pip' returned error code 1. . 

Also when i go to the pyden app (http://localhost:8000/en-GB/app/pyden/tab_home), its just a blank page