Can I add python modules to the Splunk environment?

[Simeon]

How can I add a python module that is not included in the Splunk python bundle? Specifically, I would like to use the pymssql module from within Splunk to run a scripted input.

[brwalker_ · Accepted solution]

According to the documentation you should package dependencies your app needs in its /<appname>/bin directory.

[bhavikbhalodia]

Just put all the module files in bin folder of your app and add below code to your python script and you can import any custom module:
 Let me know if you are facing any error

import os
import sys
import re

ta_name = '<splunk_app_name>'
ta_lib_name = '<parent_folder_of_python_pakage>'
pattern = re.compile(r"[\\/]etc[\\/]apps[\\/][^\\/]+[\\/]bin[\\/]?$")
new_paths = [path for path in sys.path if not pattern.search(path) or ta_name in path]
new_paths.insert(0, os.path.sep.join([ta_lib_name]))
sys.path = new_paths

Happy Splunking!!!
#dataelicitsol.com #bhavik.bhalodia@dataelicitsol.com

[bhavikbhalodia]

Just add below python code in your main script and it will work:

import os
import sys
import re

ta_name = '<splunk_app_name>'
ta_lib_name = '<parent_folder_of_python_pakage>'
pattern = re.compile(r"[\\/]etc[\\/]apps[\\/][^\\/]+[\\/]bin[\\/]?$")
new_paths = [path for path in sys.path if not pattern.search(path) or ta_name in path]
new_paths.insert(0, os.path.sep.join([ta_lib_name]))
sys.path = new_paths

[inventsekar]

Superb @bhavikbhalodia 

[bhavikbhalodia]

Thanks @inventsekar 

[badarsebard]

I've recently released a new app that allows for exactly this kind of behavior. The PyDen app allows an administrator or developer to compile different versions of Python from source and create virtual environments. Once created, you can use the custom pip command from the search bar to install packages from PyPI. Once created, you can leverage the virtual environment by copying and importing an activation script provided with the app. When the custom command you create is run, it will use the virtual environment instead of the Splunk interpreter.

The app can be found here: https://splunkbase.splunk.com/app/4322/

[kundeng]

PYDEN is archived? What's wrong with this approach? Not approved for splunk cloud? 

[thellmann]

Apps can be archived due to inactivity or per developer request. Since the last update of that app was 5 years ago, I'm assuming it was the former. 

Note that most of the suggestions in this (16-year-old) topic are considered dangerous in modern Splunk Enterprise and Splunk Cloud. The supported solution is to bundle your app's dependencies in the /bin directory of your app. Do not modify the version of Python shipped with Splunk; do not do on-stack compilation of assets your app needs; do not attempt to establish virtualization environments; etc. 

[BrunoLabelle]

Hey ebuitweb. if you put your library inside the splunk home... the minute you do an upgrade to Splunk... your libraries would get nuked... also if you have a cluster this would require you to login to every search head and edit the internal splunk home scripts directory... as well you may break the native product looking into that directory.... i would advise if you dont want headaches with support... dont put stuff in there... instead put it inside an app and deploy to the search heads. if you need more granular details about it, let me know and i can post a more granular detailed way to set it up and keep support on your side.

[ebuitweb]

Hy,
a working workharound.

on your wrapper.sh, call your python script by using full path :

/usr/bin/python $SPLUNK_HOME/bin/scripts/yourpythonscript.py

[BrunoLabelle]

although this would work technically... this would be a bad idea

Reasons:
if you add libraries in the native folder...
clustered environments would require you to login to every search heads to do this update.
you may by mistake break the native folder of splunk where it gets its own libraries.... Support wont be happy with you if they notice unwanted libraries in there.
when you upgrade Splunk software that directory gets nuked for the new release... meaning if you forget you may loose all your libraries.

To stay a happy admin and deploy quick and keep support on your side. you want to publish a hidden app with the libraries and reference them from there during execution of your code ... if your not familiar with this setup and want more details let me know and i can post a more detailed step to setting it up.

[BrunoLabelle]

Best supported way to do this and still have Splunk support on your side.
Do not overwrite Splunk native libraries as you could break the product.
or even upgrades to the product or your server could break you.
What you do is import the library on the fly during execution of your script.

Your initial script would do these three lines at the top
import sys
sys.path.insert(0,'/Path of Library you want to add')
import YOURLIBRARY

Even better for clustered environments with many search heads out there.
Create an App on your cluster master responsible for deploying to the search heads and leave it hidden so it wont show on the searcheads.
call your app lets say
"prod_searchhead_bin"
in there you have the magic folder BIN where you put all you secret libraries.

then you command would look like this
import sys
sys.path.insert(0,'/opt/splunk/etc/apps/prod_searchhead_bin/bin')
import YOURLIBRARY

then when you publish to the cluster all search heads will comply and have libraries making it a breeze during upgrades/ updates.
if you upgrade or move environments. easy to deploy as it is a native app to deploy and make all your much needed libraries available.

[joelurtubia]

Hello, I was the same problem with Mysql module that I was install on my Centos server

Splunk didn't work with this library, because splunk has they own python library...then you can fix it only added on the begin your script all libraries of python and also you must to add the python Centos library too... as this way

Find python packages

[root@xxxx]#find / -name site-packages
/usr/lib/python2.7/site-packages
/usr/lib64/python2.7/site-packages
/opt/splunk/etc/apps/Splunk_SA_Scientific_Python_linux_x86_64/bin/linux_x86_64/lib/python2.7/site-packages
/opt/splunk/lib/python2.7/site-packages

Find python binary

[root@xxxx]# whereis python
python: /usr/bin/python2.7 /usr/bin/python /usr/lib/python2.7 /usr/lib64/python2.7 /etc/python /usr/include/python2.7 /opt/splunk/bin/python /opt/splunk/bin/python2.7 /usr/share/man/man1/python.1.gz

include all at begin your script

import sys
sys.path.append('/usr/bin/python2.7')
sys.path.append('/usr/lib/python2.7/site-packages')
sys.path.append('/usr/lib64/python2.7/site-packages')

And that's it , you can run mysql module without any problem and create your alerts with this module.

Mysql Connection

import mysql.connector

I hope that this fix will help you
Joel Urtubia Ugarte

[rlacher1]

virtualenv can be used to create a standalone Python installation with whatever modules are required. The scripted input could then use a wrapper script to activate the virtualenv, execute the Python script, and deactivate the virtualenv. This works okay if the native splunk module isn't required.

script.py

#!/usr/bin/env python
import package
package.do_stuff()

script_wrapper.sh

#!/bin/bash
source /path/to/virtual/env/bin/activate
python /path/to/script.py
deactivate

[asifhj]

Guys,

Its simple,
Copy your required packages/files from

---

/usr/local/lib/python2.7/dist-packages

to

/opt/splunk/lib/python2.7/site-packages/

---

[hans]

I just want to add how i installed ipython on splunk's python install. I downloaded the tar.gz and did:

$ splunk cmd python setup.py install

I got an error:

ImportError: No module named command.build_py

It turns out distutils in splunk's python install does not have the command module. The way I got around this is to use distutils from another python 2.7 install by adding its directory to PYTHONPATH:

$ export PYTHONPATH=$HOME/python272/lib/python2.7

and installed using setup.py again

$ splunk cmd python setup.py install

finally:

$ splunk cmd ipython

[araitz]

Here is an example, tested on Linux and Windows, that uses a non-splunk python but still lets you load most Splunk modules:

wrapper.py:

import os
import subprocess

_NEW_PYTHON_PATH = '/usr/bin/python'
_SPLUNK_PYTHON_PATH = os.environ['PYTHONPATH']

os.environ['PYTHONPATH'] = _NEW_PYTHON_PATH 
my_process = os.path.join(os.getcwd(), 'my_script.py')

p = subprocess.Popen([os.environ['PYTHONPATH'], my_process, _SPLUNK_PYTHON_PATH], 
                      stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
output = p.communicate()[0]
print output

my_script.py:

import sys 
import os

#This step is necessary in order to load splunk packages
from optparse import OptionParser
parser = OptionParser()
(options, args) = parser.parse_args()
_SPLUNK_PYTHON_PATH = args[0]
sys.path.append(_SPLUNK_PYTHON_PATH)

import splunk
import cherrypy
import some_package_from_new_python
...

[jrouse025]

Here is a full example @araitz thanks for the inspiration
link text

[heshamzaid]

how to make this on django web framework
i make an splunk app and i need to import some non native python module

[the_wolverine]

Updating your Splunk Python is not supported (by Splunk). The recommended method is to update your system's Python or install a 3rd copy of Python that is not located in your PATH.