There's also a more upgrade-friendly way to accomplish this. Some of our users setup whichever script they've configured in Splunk as a pass-through to a script that runs using their system Python (with whichever custom modules they've installed).
The steps are roughly:
configure your script in splunk (search script, scripted input, whatever)
this script should:
unset PYTHONPATH (in os.environ)
perhaps unset LD_LIBRARY_PATH, depending on your environment (also in os.environ)
create a process to run /usr/bin/python (via subprocess)
redirect stdin, stdout, stderr to/from script2
script2 can then load any arbitrary python module installed in your system's python installation.
Just put all the module files in bin folder of your app and add below code to your python script and you can import any custom module:
Let me know if you are facing any error
import os import sys import re ta_name = '<splunk_app_name>' ta_lib_name = '<parent_folder_of_python_pakage>' pattern = re.compile(r"[\\/]etc[\\/]apps[\\/][^\\/]+[\\/]bin[\\/]?$") new_paths = [path for path in sys.path if not pattern.search(path) or ta_name in path] new_paths.insert(0, os.path.sep.join([ta_lib_name])) sys.path = new_paths
Happy Splunking!!!
#dataelicitsol.com #bhavik.bhalodia@dataelicitsol.com
Just add below python code in your main script and it will work:
import os
import sys
import re
ta_name = '<splunk_app_name>'
ta_lib_name = '<parent_folder_of_python_pakage>'
pattern = re.compile(r"[\\/]etc[\\/]apps[\\/][^\\/]+[\\/]bin[\\/]?$")
new_paths = [path for path in sys.path if not pattern.search(path) or ta_name in path]
new_paths.insert(0, os.path.sep.join([ta_lib_name]))
sys.path = new_paths
Superb @bhavikbhalodia
Thanks @inventsekar
I've recently released a new app that allows for exactly this kind of behavior. The PyDen app allows an administrator or developer to compile different versions of Python from source and create virtual environments. Once created, you can use the custom pip command from the search bar to install packages from PyPI. Once created, you can leverage the virtual environment by copying and importing an activation script provided with the app. When the custom command you create is run, it will use the virtual environment instead of the Splunk interpreter.
The app can be found here: https://splunkbase.splunk.com/app/4322/
Hey ebuitweb. if you put your library inside the splunk home... the minute you do an upgrade to Splunk... your libraries would get nuked... also if you have a cluster this would require you to login to every search head and edit the internal splunk home scripts directory... as well you may break the native product looking into that directory.... i would advise if you dont want headaches with support... dont put stuff in there... instead put it inside an app and deploy to the search heads. if you need more granular details about it, let me know and i can post a more granular detailed way to set it up and keep support on your side.
Hy,
a working workharound.
on your wrapper.sh, call your python script by using full path :
/usr/bin/python $SPLUNK_HOME/bin/scripts/yourpythonscript.py
although this would work technically... this would be a bad idea
Reasons:
if you add libraries in the native folder...
clustered environments would require you to login to every search heads to do this update.
you may by mistake break the native folder of splunk where it gets its own libraries.... Support wont be happy with you if they notice unwanted libraries in there.
when you upgrade Splunk software that directory gets nuked for the new release... meaning if you forget you may loose all your libraries.
To stay a happy admin and deploy quick and keep support on your side. you want to publish a hidden app with the libraries and reference them from there during execution of your code ... if your not familiar with this setup and want more details let me know and i can post a more detailed step to setting it up.
Best supported way to do this and still have Splunk support on your side.
Do not overwrite Splunk native libraries as you could break the product.
or even upgrades to the product or your server could break you.
What you do is import the library on the fly during execution of your script.
Your initial script would do these three lines at the top
import sys
sys.path.insert(0,'/Path of Library you want to add')
import YOURLIBRARY
Even better for clustered environments with many search heads out there.
Create an App on your cluster master responsible for deploying to the search heads and leave it hidden so it wont show on the searcheads.
call your app lets say
"prod_searchhead_bin"
in there you have the magic folder BIN where you put all you secret libraries.
then you command would look like this
import sys
sys.path.insert(0,'/opt/splunk/etc/apps/prod_searchhead_bin/bin')
import YOURLIBRARY
then when you publish to the cluster all search heads will comply and have libraries making it a breeze during upgrades/ updates.
if you upgrade or move environments. easy to deploy as it is a native app to deploy and make all your much needed libraries available.
Hello, I was the same problem with Mysql module that I was install on my Centos server
Splunk didn't work with this library, because splunk has they own python library...then you can fix it only added on the begin your script all libraries of python and also you must to add the python Centos library too... as this way
[root@xxxx]#find / -name site-packages
/usr/lib/python2.7/site-packages
/usr/lib64/python2.7/site-packages
/opt/splunk/etc/apps/Splunk_SA_Scientific_Python_linux_x86_64/bin/linux_x86_64/lib/python2.7/site-packages
/opt/splunk/lib/python2.7/site-packages
[root@xxxx]# whereis python
python: /usr/bin/python2.7 /usr/bin/python /usr/lib/python2.7 /usr/lib64/python2.7 /etc/python /usr/include/python2.7 /opt/splunk/bin/python /opt/splunk/bin/python2.7 /usr/share/man/man1/python.1.gz
include all at begin your script
import sys
sys.path.append('/usr/bin/python2.7')
sys.path.append('/usr/lib/python2.7/site-packages')
sys.path.append('/usr/lib64/python2.7/site-packages')
And that's it , you can run mysql module without any problem and create your alerts with this module.
import mysql.connector
I hope that this fix will help you
Joel Urtubia Ugarte
virtualenv can be used to create a standalone Python installation with whatever modules are required. The scripted input could then use a wrapper script to activate the virtualenv, execute the Python script, and deactivate the virtualenv. This works okay if the native splunk module isn't required.
script.py
#!/usr/bin/env python
import package
package.do_stuff()
script_wrapper.sh
#!/bin/bash
source /path/to/virtual/env/bin/activate
python /path/to/script.py
deactivate
Guys,
Its simple,
Copy your required packages/files from
/usr/local/lib/python2.7/dist-packages
to
/opt/splunk/lib/python2.7/site-packages/
I just want to add how i installed ipython
on splunk's python install. I downloaded the tar.gz and did:
$ splunk cmd python setup.py install
I got an error:
ImportError: No module named command.build_py
It turns out distutils
in splunk's python install does not have the command module. The way I got around this is to use distutils
from another python 2.7 install by adding its directory to PYTHONPATH
:
$ export PYTHONPATH=$HOME/python272/lib/python2.7
and installed using setup.py again
$ splunk cmd python setup.py install
finally:
$ splunk cmd ipython
Here is an example, tested on Linux and Windows, that uses a non-splunk python but still lets you load most Splunk modules:
wrapper.py:
import os
import subprocess
_NEW_PYTHON_PATH = '/usr/bin/python'
_SPLUNK_PYTHON_PATH = os.environ['PYTHONPATH']
os.environ['PYTHONPATH'] = _NEW_PYTHON_PATH
my_process = os.path.join(os.getcwd(), 'my_script.py')
p = subprocess.Popen([os.environ['PYTHONPATH'], my_process, _SPLUNK_PYTHON_PATH],
stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
output = p.communicate()[0]
print output
my_script.py:
import sys
import os
#This step is necessary in order to load splunk packages
from optparse import OptionParser
parser = OptionParser()
(options, args) = parser.parse_args()
_SPLUNK_PYTHON_PATH = args[0]
sys.path.append(_SPLUNK_PYTHON_PATH)
import splunk
import cherrypy
import some_package_from_new_python
...
Here is a full example @araitz thanks for the inspiration
link text
how to make this on django web framework
i make an splunk app and i need to import some non native python module
Updating your Splunk Python is not supported (by Splunk). The recommended method is to update your system's Python or install a 3rd copy of Python that is not located in your PATH.
No, there's no way to tell splunk to run some other python. However, you can make the script itself do this as per amrit's comment.
i have 3 pythons in my linux server, local 2.3, 2.5 and 2.6(in splunk4.0.8). I can not install MySQL-python-1.2.3c1 to 2.6, but it's ok to install it to local 2.3. the following is the detail.
cd MySQL-python-1.2.3c1
python2.6 setup.py build ImportError: No module named setuptools
sh setuptools-0.6c11-py2.6.egg ImportError: No module named command.bdist
[root@localhost bin]# ./splunk cmd /local/dl/setuptools-0.6c11-py2.6.egg Traceback (most recent call last): File "", line 1, in File "/local/dl/setuptools-0.6c11-py2.6.egg/setuptools/init.py", line 2, in File "/local/dl/setuptools-0.6c11-py2.6.egg/setuptools/extension.py", line 2, in File "/local/dl/setuptools-0.6c11-py2.6.egg/setuptools/dist.py", line 5, in File "/local/dl/setuptools-0.6c11-py2.6.egg/setuptools/command/init.py", line 13, in ImportError: No module named command.bdist
[root@localhost bin]# ./splunk cmd /local/dl/MySQL_python-1.2.3c1-py2.6-linux-i686.egg couldn't run "/local/dl/MySQL_python-1.2.3c1-py2.6-linux-i686.egg": Exec format error
@the_wolverine, how to tell splunk to execute lookup python script with system's Python?
Thanks,
Dianbo
There's also a more upgrade-friendly way to accomplish this. Some of our users setup whichever script they've configured in Splunk as a pass-through to a script that runs using their system Python (with whichever custom modules they've installed).
The steps are roughly:
configure your script in splunk (search script, scripted input, whatever)
this script should:
unset PYTHONPATH (in os.environ)
perhaps unset LD_LIBRARY_PATH, depending on your environment (also in os.environ)
create a process to run /usr/bin/python (via subprocess)
redirect stdin, stdout, stderr to/from script2
script2 can then load any arbitrary python module installed in your system's python installation.