topic Re: Import python libraries into script - things don't work inside Splunk that work on Command line... in Splunk Dev

Import python libraries into script - things don't work inside Splunk that work on Command line...

phil__tanner — Sun, 07 Jun 2020 18:08:42 GMT

TL;DR - the python scripts runing from the $SPLUNK_HOME$/bin/splunk cmd python throw errors that aren't thrown using /usr/bin/python.

So, I've got a load of data which has X/Y coordinates stored in NZTM2000 format, and I want to enrich the data by converting to a Lat/Lng so that I can plot onto a map visualisation, or output to other systems that use this format. I've created an app (replaced with in the examples below), and dropped a Python script file in there as a proof of concept.

The Python script I'm using to convert requires a couple of additional libraries (installed using "pip install pyproj") which I installed as the Splunk user (who's home directory is also set to $SPLUNK_HOME$).

The problem is that the script works fine from the command line:

splunk@standalone-vm:~/etc/apps/<appname>/bin$ /usr/bin/python /opt/splunk/etc/apps/<appname>/bin/nztm_to_lat_lng.py
(-40.981429307189785, 174.95992442484228)

However, if I try running it using the Splunk binary route, I get an error....

splunk@standalone-vm:~/etc/apps/<appname>/bin$ /opt/splunk/bin/splunk cmd python /opt/splunk/etc/apps/<appname>/bin/nztm_to_lat_lng.py
Traceback (most recent call last):
  File "/opt/splunk/etc/apps/<appname>/bin/nztm_to_lat_lng.py", line 14, in <module>
    from pyproj import Proj, transform
  File "/opt/splunk/.local/lib/python2.7/site-packages/pyproj/__init__.py", line 69, in <module>
    from pyproj._list import (  # noqa: F401
ImportError: /opt/splunk/.local/lib/python2.7/site-packages/pyproj/_list.so: undefined symbol: PyUnicodeUCS4_FromStringAndSize

So I thought it might be something to do with the 2.7 version, but it seems not:

splunk@standalone-vm:~/etc/apps/<appname>/bin$ /usr/bin/python --version
Python 2.7.17
splunk@standalone-vm:~/etc/apps/<appname>/bin$ /opt/splunk/bin/splunk cmd python --version
Python 2.7.17

Which seems to rule that out. So, I tried a different route:

splunk@standalone-vm:~/etc/apps/<appname>/bin$ /opt/splunk/bin/splunk cmd /usr/bin/python /opt/splunk/etc/apps/<appname>/bin/nztm_to_lat_lng.py
(-40.981429307189785, 174.95992442484228)

So if I get Splunk to call the /usr/bin/python route, everything works as expected. So this points to a problem with the shipped version of python shipped with Splunk, but that seems.... Unlikely.

So, help, please?

Appendix: Here's a copy of the python file I'm trying to run (Note: you'll need the pyproj libary too...)

#!/usr/bin/env python

"""nztm_to_lat_lng.py: Takes an NZTM2000 X/Y coordinate system and outputs a lat/lng"""
__author__     = "Phil Tanner"
__version__    = "0.0.1"
__status__     = "Prototype"

# If we're running from Splunk command, patch our local path so we can load modules; hack!
import os
splunk_home = os.getenv('SPLUNK_HOME')
if splunk_home is not None:
  import sys
  # Local app path
  sys.path.append(os.path.join(os.environ['SPLUNK_HOME'],'etc','apps','<appname>','bin'))
  # Splunk user path - note this is fixed to the 2.7 to pull in the pip installed pyproj files, without which it fails horribly
  sys.path.append(os.path.join(os.environ['SPLUNK_HOME'],'.local','lib','python2.7','site-packages'))

from pyproj import Proj, transform
inProj = Proj("+proj=tmerc +lat_0=0 +lon_0=173 +k=0.9996 +x_0=1600000 +y_0=10000000 +ellps=GRS80 +towgs84=0,0,0,0,0,0,0 +units=m +no_defs")
outProj = Proj('epsg:4326')

# Note: hardcoded sample, literally chosen at random!
x1,y1 = 1764883.4618,5461454.6101
# Convert between NZTM & Lat/Lng
x2,y2 = transform(inProj,outProj,x1,y1)

print (x2,y2)

And for reference, here's the Linux version I'm running the server on (which is the standard MS Azure Splunk instance):

splunk@standalone-vm:~/etc/apps/<appname>/bin$ uname -a
Linux standalone-vm 4.15.0-96-generic #97-Ubuntu SMP Wed Apr 1 03:25:46 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

Re: Import python libraries into script - things don't work inside Splunk that work on Command line...

MuS — Thu, 21 May 2020 05:50:34 GMT

Hi phil__tanner,

from what I experienced so far it's the best, easiest to put the module into the $SPLUNK_HOME/etc/apps/<appname>/bin directory and import it from there.

Hope this helps ...

cheers, MuS

PS: Greetings up the coast 😉

Re: Import python libraries into script - things don't work inside Splunk that work on Command line...

phil__tanner — Thu, 21 May 2020 20:56:05 GMT

Hey MuS.

Tried replying last night, got lots of 500 errors.

I should've mentioned, I tried copying my repos gained from ~splunk/.local/lib/python2.7/site-packages/ into the <appname>/bin directory too. Gave me the same error as bringing them in via the path (unsurprisingly) - i.e.:

Traceback (most recent call last):
  File "/opt/splunk/etc/apps/<appname>/bin/nztm_to_lat_lng.py", line 8, in <module>
    from pyproj import Proj, transform
  File "/opt/splunk/etc/apps/<appname>/bin/pyproj/__init__.py", line 69, in <module>
    from pyproj._list import (  # noqa: F401
ImportError: /opt/splunk/etc/apps/<appname>/bin/pyproj/_list.so: undefined symbol: PyUnicodeUCS4_FromStringAndSize

I had thought it might be an incompatibility (32/64 bit?) between the python shipped with Splunk & the python on the server which pip is compiling packages against. But the output of these commands seems to indicate a 64bit running in each case:

splunk@standalone-vm:~$ /usr/bin/python -c 'import sys;print("%x" % sys.maxsize, sys.maxsize > 2**32)'
('7fffffffffffffff', True)
splunk@standalone-vm:~$ /opt/splunk/bin/splunk cmd python -c 'import sys;print("%x" % sys.maxsize, sys.maxsize > 2**32)'
('7fffffffffffffff', True)

(Test results based on advice in this post: https://stackoverflow.com/questions/1405913/how-do-i-determine-if-my-python-shell-is-executing-in-32bit-or-64bit-mode-on-os )

Thanks, Phil

PS: Greetings from the flat lands into the hills & valleys the Bay.

Re: Import python libraries into script - things don't work inside Splunk that work on Command line...

MuS — Mon, 25 May 2020 22:21:27 GMT

My guess you hit this problem here https://stackoverflow.com/a/33796439

cheers, MuS

Re: Import python libraries into script - things don't work inside Splunk that work on Command line...

phil__tanner — Tue, 26 May 2020 00:21:50 GMT

Yeah, I ended up at that same place.

Does this mean that I should be replacing the python binary in $SPLUNK_HOME$/bin/python with a differently compiled version? Because the pip install I ran as the splunk user clearly built against the OS system of python, not the Splunk version, which is where the conflict first originated, and Splunk doesn't seem to ship with a pip version I could compile against....

Any which way, it looks like this isn't going to be a "upgrade proof" solution?

Re: Import python libraries into script - things don't work inside Splunk that work on Command line...

MuS — Tue, 26 May 2020 01:04:21 GMT

I would not recommend to replace the shipped python that will result in a lot of tears 😉
Like you said, anything you build will not be upgrade proof

cheers, MuS

Re: Import python libraries into script - things don't work inside Splunk that work on Command line.

livehybrid — Wed, 10 Jun 2020 21:54:47 GMT

You could try installing the package locally to the script to see if that makes a difference?

pip install -t . pyproj