topic Re: Can I add python modules to the Splunk environment? in Splunk Dev

Can I add python modules to the Splunk environment?

Simeon — Thu, 18 Jun 2020 18:14:32 GMT

How can I add a python module that is not included in the Splunk python bundle? Specifically, I would like to use the pymssql module from within Splunk to run a scripted input.

Re: Can I add python modules to the Splunk environment?

igor — Sat, 07 Nov 2009 04:37:32 GMT

Just use the standard way of installing modules (make sure that splunk and python are in path):
Untar pymssql-1.0.2.tar.gz
cd pymssql-1.0.2
splunk cmd python setup.py install

Re: Can I add python modules to the Splunk environment?

Johnvey — Sat, 07 Nov 2009 06:23:15 GMT

Users are free to install any python module they desire. The caveats are, 1) upgrading Splunk may break them, 2) installing newer versions of packages that come with Splunk may produce unknown interaction problems.

To install a python package in splunk:

$ splunk cmd <python_install_command>

So if the package uses the setup.py method:

$ cd path_to_package_setup
$ splunk cmd python setup.py install

Or if it's an egg,

$ splunk cmd ./my-python-installer.egg

The final location of the installed modules would be:

$SPLUNK_HOME/lib/python2.6/site-packages

Re: Can I add python modules to the Splunk environment?

amrit — Fri, 13 Nov 2009 03:06:21 GMT

There's also a more upgrade-friendly way to accomplish this. Some of our users setup whichever script they've configured in Splunk as a pass-through to a script that runs using their system Python (with whichever custom modules they've installed).

The steps are roughly:

configure your script in splunk (search script, scripted input, whatever)
this script should:
- unset PYTHONPATH (in os.environ)
- perhaps unset LD_LIBRARY_PATH, depending on your environment (also in os.environ)
- create a process to run /usr/bin/python (via subprocess)
- redirect stdin, stdout, stderr to/from script2

script2 can then load any arbitrary python module installed in your system's python installation.

Re: Can I add python modules to the Splunk environment?

the_wolverine — Sat, 13 Feb 2010 06:34:14 GMT

Updating your Splunk Python is not supported (by Splunk). The recommended method is to update your system's Python or install a 3rd copy of Python that is not located in your PATH.

Re: Can I add python modules to the Splunk environment?

dianbo — Thu, 11 Mar 2010 11:00:09 GMT

i have 3 pythons in my linux server, local 2.3, 2.5 and 2.6(in splunk4.0.8). I can not install MySQL-python-1.2.3c1 to 2.6, but it's ok to install it to local 2.3. the following is the detail.

cd MySQL-python-1.2.3c1

python2.6 setup.py build ImportError: No module named setuptools

sh setuptools-0.6c11-py2.6.egg ImportError: No module named command.bdist

[root@localhost bin]# ./splunk cmd /local/dl/setuptools-0.6c11-py2.6.egg Traceback (most recent call last): File "", line 1, in File "/local/dl/setuptools-0.6c11-py2.6.egg/setuptools/init.py", line 2, in File "/local/dl/setuptools-0.6c11-py2.6.egg/setuptools/extension.py", line 2, in File "/local/dl/setuptools-0.6c11-py2.6.egg/setuptools/dist.py", line 5, in File "/local/dl/setuptools-0.6c11-py2.6.egg/setuptools/command/init.py", line 13, in ImportError: No module named command.bdist

[root@localhost bin]# ./splunk cmd /local/dl/MySQL_python-1.2.3c1-py2.6-linux-i686.egg couldn't run "/local/dl/MySQL_python-1.2.3c1-py2.6-linux-i686.egg": Exec format error

@the_wolverine, how to tell splunk to execute lookup python script with system's Python?

Thanks,

Dianbo

Re: Can I add python modules to the Splunk environment?

jrodman — Thu, 16 Dec 2010 02:44:46 GMT

No, there's no way to tell splunk to run some other python. However, you can make the script itself do this as per amrit's comment.

Re: Can I add python modules to the Splunk environment?

wollinet — Sat, 29 Jan 2011 00:21:55 GMT

Is that still supported with 4.1 ? I tried both ways, first one prouced errors, second one didn't produce any output and nothing was installed.

Re: Can I add python modules to the Splunk environment?

araitz — Thu, 03 Feb 2011 06:17:13 GMT

Here is an example, tested on Linux and Windows, that uses a non-splunk python but still lets you load most Splunk modules:

wrapper.py:

import os
import subprocess

_NEW_PYTHON_PATH = '/usr/bin/python'
_SPLUNK_PYTHON_PATH = os.environ['PYTHONPATH']

os.environ['PYTHONPATH'] = _NEW_PYTHON_PATH 
my_process = os.path.join(os.getcwd(), 'my_script.py')

p = subprocess.Popen([os.environ['PYTHONPATH'], my_process, _SPLUNK_PYTHON_PATH], 
                      stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
output = p.communicate()[0]
print output

my_script.py:

import sys 
import os

#This step is necessary in order to load splunk packages
from optparse import OptionParser
parser = OptionParser()
(options, args) = parser.parse_args()
_SPLUNK_PYTHON_PATH = args[0]
sys.path.append(_SPLUNK_PYTHON_PATH)

import splunk
import cherrypy
import some_package_from_new_python
...

Re: Can I add python modules to the Splunk environment?

Dark_Ichigo — Tue, 04 Oct 2011 07:27:12 GMT

Is there a step by step tutorial to accomplish this on a Linux environment?

Re: Can I add python modules to the Splunk environment?

hans — Mon, 19 Mar 2012 15:56:10 GMT

I just want to add how i installed ipython on splunk's python install. I downloaded the tar.gz and did:

$ splunk cmd python setup.py install

I got an error:

ImportError: No module named command.build_py

It turns out distutils in splunk's python install does not have the command module. The way I got around this is to use distutils from another python 2.7 install by adding its directory to PYTHONPATH:

$ export PYTHONPATH=$HOME/python272/lib/python2.7

and installed using setup.py again

$ splunk cmd python setup.py install

finally:

$ splunk cmd ipython

Re: Can I add python modules to the Splunk environment?

asifhj — Fri, 27 Sep 2013 14:13:47 GMT

Guys,

Its simple,
Copy your required packages/files from

/usr/local/lib/python2.7/dist-packages

to

/opt/splunk/lib/python2.7/site-packages/

Re: Can I add python modules to the Splunk environment?

bmacias84 — Tue, 07 Jul 2015 17:25:30 GMT

As another option I would checkout this post on how to add egg file or source files while still using Splunk python rather than system. http://answers.splunk.com/answers/220196/import-non-native-python-libraries-into-splunk.html#answer-222585

Re: Can I add python modules to the Splunk environment?

rlacher1 — Tue, 17 May 2016 15:28:39 GMT

virtualenv can be used to create a standalone Python installation with whatever modules are required. The scripted input could then use a wrapper script to activate the virtualenv, execute the Python script, and deactivate the virtualenv. This works okay if the native splunk module isn't required.

script.py

#!/usr/bin/env python
import package
package.do_stuff()

script_wrapper.sh

#!/bin/bash
source /path/to/virtual/env/bin/activate
python /path/to/script.py
deactivate

Re: Can I add python modules to the Splunk environment?

heshamzaid — Tue, 03 Jan 2017 15:29:35 GMT

how to make this on django web framework
i make an splunk app and i need to import some non native python module

Re: Can I add python modules to the Splunk environment?

joelurtubia — Tue, 29 Sep 2020 16:36:38 GMT

Hello, I was the same problem with Mysql module that I was install on my Centos server

Splunk didn't work with this library, because splunk has they own python library...then you can fix it only added on the begin your script all libraries of python and also you must to add the python Centos library too... as this way

Find python packages

[root@xxxx]#find / -name site-packages
/usr/lib/python2.7/site-packages
/usr/lib64/python2.7/site-packages
/opt/splunk/etc/apps/Splunk_SA_Scientific_Python_linux_x86_64/bin/linux_x86_64/lib/python2.7/site-packages
/opt/splunk/lib/python2.7/site-packages

Find python binary

[root@xxxx]# whereis python
python: /usr/bin/python2.7 /usr/bin/python /usr/lib/python2.7 /usr/lib64/python2.7 /etc/python /usr/include/python2.7 /opt/splunk/bin/python /opt/splunk/bin/python2.7 /usr/share/man/man1/python.1.gz

include all at begin your script

import sys
sys.path.append('/usr/bin/python2.7')
sys.path.append('/usr/lib/python2.7/site-packages')
sys.path.append('/usr/lib64/python2.7/site-packages')

And that's it , you can run mysql module without any problem and create your alerts with this module.

Mysql Connection

import mysql.connector

I hope that this fix will help you
Joel Urtubia Ugarte

Re: Can I add python modules to the Splunk environment?

BrunoLabelle — Tue, 29 Sep 2020 19:44:54 GMT

Best supported way to do this and still have Splunk support on your side.
Do not overwrite Splunk native libraries as you could break the product.
or even upgrades to the product or your server could break you.
What you do is import the library on the fly during execution of your script.

Your initial script would do these three lines at the top
import sys
sys.path.insert(0,'/Path of Library you want to add')
import YOURLIBRARY

Even better for clustered environments with many search heads out there.
Create an App on your cluster master responsible for deploying to the search heads and leave it hidden so it wont show on the searcheads.
call your app lets say
"prod_searchhead_bin"
in there you have the magic folder BIN where you put all you secret libraries.

then you command would look like this
import sys
sys.path.insert(0,'/opt/splunk/etc/apps/prod_searchhead_bin/bin')
import YOURLIBRARY

then when you publish to the cluster all search heads will comply and have libraries making it a breeze during upgrades/ updates.
if you upgrade or move environments. easy to deploy as it is a native app to deploy and make all your much needed libraries available.

Re: Can I add python modules to the Splunk environment?

ebuitweb — Thu, 08 Nov 2018 13:20:01 GMT

Hy,
a working workharound.

on your wrapper.sh, call your python script by using full path :

/usr/bin/python $SPLUNK_HOME/bin/scripts/yourpythonscript.py

Re: Can I add python modules to the Splunk environment?

BrunoLabelle — Thu, 08 Nov 2018 18:49:08 GMT

Hey ebuitweb. if you put your library inside the splunk home... the minute you do an upgrade to Splunk... your libraries would get nuked... also if you have a cluster this would require you to login to every search head and edit the internal splunk home scripts directory... as well you may break the native product looking into that directory.... i would advise if you dont want headaches with support... dont put stuff in there... instead put it inside an app and deploy to the search heads. if you need more granular details about it, let me know and i can post a more granular detailed way to set it up and keep support on your side.

Re: Can I add python modules to the Splunk environment?

BrunoLabelle — Thu, 08 Nov 2018 18:54:05 GMT

although this would work technically... this would be a bad idea

Reasons:
if you add libraries in the native folder...
clustered environments would require you to login to every search heads to do this update.
you may by mistake break the native folder of splunk where it gets its own libraries.... Support wont be happy with you if they notice unwanted libraries in there.
when you upgrade Splunk software that directory gets nuked for the new release... meaning if you forget you may loose all your libraries.

To stay a happy admin and deploy quick and keep support on your side. you want to publish a hidden app with the libraries and reference them from there during execution of your code ... if your not familiar with this setup and want more details let me know and i can post a more detailed step to setting it up.