TL;DR - the python scripts runing from the $SPLUNK_HOME$/bin/splunk cmd python throw errors that aren't thrown using /usr/bin/python.
So, I've got a load of data which has X/Y coordinates stored...
See more...
TL;DR - the python scripts runing from the $SPLUNK_HOME$/bin/splunk cmd python throw errors that aren't thrown using /usr/bin/python.
So, I've got a load of data which has X/Y coordinates stored in NZTM2000 format, and I want to enrich the data by converting to a Lat/Lng so that I can plot onto a map visualisation, or output to other systems that use this format. I've created an app (replaced with in the examples below), and dropped a Python script file in there as a proof of concept.
The Python script I'm using to convert requires a couple of additional libraries (installed using "pip install pyproj") which I installed as the Splunk user (who's home directory is also set to $SPLUNK_HOME$).
The problem is that the script works fine from the command line:
splunk@standalone-vm:~/etc/apps/<appname>/bin$ /usr/bin/python /opt/splunk/etc/apps/<appname>/bin/nztm_to_lat_lng.py
(-40.981429307189785, 174.95992442484228)
However, if I try running it using the Splunk binary route, I get an error....
splunk@standalone-vm:~/etc/apps/<appname>/bin$ /opt/splunk/bin/splunk cmd python /opt/splunk/etc/apps/<appname>/bin/nztm_to_lat_lng.py
Traceback (most recent call last):
File "/opt/splunk/etc/apps/<appname>/bin/nztm_to_lat_lng.py", line 14, in <module>
from pyproj import Proj, transform
File "/opt/splunk/.local/lib/python2.7/site-packages/pyproj/__init__.py", line 69, in <module>
from pyproj._list import ( # noqa: F401
ImportError: /opt/splunk/.local/lib/python2.7/site-packages/pyproj/_list.so: undefined symbol: PyUnicodeUCS4_FromStringAndSize
So I thought it might be something to do with the 2.7 version, but it seems not:
splunk@standalone-vm:~/etc/apps/<appname>/bin$ /usr/bin/python --version
Python 2.7.17
splunk@standalone-vm:~/etc/apps/<appname>/bin$ /opt/splunk/bin/splunk cmd python --version
Python 2.7.17
Which seems to rule that out. So, I tried a different route:
splunk@standalone-vm:~/etc/apps/<appname>/bin$ /opt/splunk/bin/splunk cmd /usr/bin/python /opt/splunk/etc/apps/<appname>/bin/nztm_to_lat_lng.py
(-40.981429307189785, 174.95992442484228)
So if I get Splunk to call the /usr/bin/python route, everything works as expected. So this points to a problem with the shipped version of python shipped with Splunk, but that seems.... Unlikely.
So, help, please?
Appendix: Here's a copy of the python file I'm trying to run (Note: you'll need the pyproj libary too...)
#!/usr/bin/env python
"""nztm_to_lat_lng.py: Takes an NZTM2000 X/Y coordinate system and outputs a lat/lng"""
__author__ = "Phil Tanner"
__version__ = "0.0.1"
__status__ = "Prototype"
# If we're running from Splunk command, patch our local path so we can load modules; hack!
import os
splunk_home = os.getenv('SPLUNK_HOME')
if splunk_home is not None:
import sys
# Local app path
sys.path.append(os.path.join(os.environ['SPLUNK_HOME'],'etc','apps','<appname>','bin'))
# Splunk user path - note this is fixed to the 2.7 to pull in the pip installed pyproj files, without which it fails horribly
sys.path.append(os.path.join(os.environ['SPLUNK_HOME'],'.local','lib','python2.7','site-packages'))
from pyproj import Proj, transform
inProj = Proj("+proj=tmerc +lat_0=0 +lon_0=173 +k=0.9996 +x_0=1600000 +y_0=10000000 +ellps=GRS80 +towgs84=0,0,0,0,0,0,0 +units=m +no_defs")
outProj = Proj('epsg:4326')
# Note: hardcoded sample, literally chosen at random!
x1,y1 = 1764883.4618,5461454.6101
# Convert between NZTM & Lat/Lng
x2,y2 = transform(inProj,outProj,x1,y1)
print (x2,y2)
And for reference, here's the Linux version I'm running the server on (which is the standard MS Azure Splunk instance):
splunk@standalone-vm:~/etc/apps/<appname>/bin$ uname -a
Linux standalone-vm 4.15.0-96-generic #97-Ubuntu SMP Wed Apr 1 03:25:46 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux