Adding Python Module to Splunk

[kholleran]

Hello,

I am trying to add the active_directory module to Splunk Python so I can query OU's for specific users to pass into a search.

I found a similar question here: http://answers.splunk.com/questions/8/can-i-add-python-modules-to-the-splunk-environment

However I when I do a

splunk cmd python module_dir\setup.py install 

I receive the following error:

usage: setup.py [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
   or: setup.py --help [cmd1 cmd2 ...]
   or: setup.py --help-commands
   or: setup.py cmd --help

error: invalid command 'install'

Thanks for any help.

Kevin

[Stephen_Sorkin · Accepted solution]

I tried this as well and encountered the same problem. We'll track it down and see the root cause, but there's an easy workaround that's preferable, especially if you're putting this search command into a separate app.

The active_directory python module appears to be just a single .py file. If you put this file in $SPLUNK_HOME/etc/apps/<app>/bin/, you should be able to import it from your search command. This technique should work with any pure python module (that doesn't require native C/C++ code to be compiled). You can see an example of this in the geoip app on splunkbase: http://www.splunkbase.com/apps/All/4.x/Add-On/app:Geo+Location+Lookup+Script. In that particular case a directory is placed in the app's bin directory.

[cneberg]

If you are just going to copy the distutils directory from another installation of python, make sure it is the same sub-sub-version of python as well 2.6.4 vs 2.6.5, etc. It was easier to build a new copy of python on the same system of the exact same version as the splunk version, then do builds of python modules there. Then copy the generated shared libraries and .py files into the splunk version of python in the equivalent locations.

[kholleran]

Thanks for your help. Putting the py file in the bin worked just fine. I will try the second suggestion when I get time as this sounds like it would be a long term solution for other modules.

Thanks!

[joelurtubia]

Hello, I was the same problem with Mysql module that I was install on my Centos server

Splunk didn't work with this library, because splunk has they own python library...then you can fix it only added on the begin your script all libraries of python and also you must to add the python Centos library too... as this way

Find python packages

[root@xxxx]#find / -name site-packages
/usr/lib/python2.7/site-packages
/usr/lib64/python2.7/site-packages
/opt/splunk/etc/apps/Splunk_SA_Scientific_Python_linux_x86_64/bin/linux_x86_64/lib/python2.7/site-packages
/opt/splunk/lib/python2.7/site-packages

Find python binary

[root@xxxx]# whereis python
python: /usr/bin/python2.7 /usr/bin/python /usr/lib/python2.7 /usr/lib64/python2.7 /etc/python /usr/include/python2.7 /opt/splunk/bin/python /opt/splunk/bin/python2.7 /usr/share/man/man1/python.1.gz

include all at begin your script

import sys
sys.path.append('/usr/bin/python2.7')
sys.path.append('/usr/lib/python2.7/site-packages')
sys.path.append('/usr/lib64/python2.7/site-packages')

And that's it , you can run mysql module without any problem and create your alerts with this module.

Mysql Connection

import mysql.connector

I hope that this fix will help you
Joel Urtubia Ugarte

[Lowell · Accepted solution]

The distutils shipped with Splunk's bundled python seems to be the minimal version which does not allow building of new python modules out of the box.

I "solved" this on my system by simply replacing the $SPLUNK_HOME/lib/python2.6/distutils folder with my locally installed copy; which will probably be at a path like: /usr/lib/python26/distutils (You probably only want to do this with the same version of python, 2.6 in this example, but I'm not sure.) Note: I did have to keep the old copy of sysconfig.py around, which must have been different between the two python installs.

Also note that you may have to install this package separately. On Ubuntu, this can be done with the command apt-get install python-dev. The standard windows install I think should include the full distutils package.

I really think there should be a "better" (or at least) fully documented way to do this.

---

It appears that Splunk ships with the standard minimal Distutils that comes packaged in the standard python source code distribution. You is confirmed in the file $SPLUNK_HOME/lib/python2.6/distutils/README, which says:

This directory contains only a subset of the Distutils, specifically
the Python modules in the 'distutils' and 'distutils.command'
packages. This is all you need to distribute and install Python
modules using the Distutils. There is also a separately packaged
standalone version of the Distutils available for people who want to
upgrade the Distutils without upgrading Python, available from the
Distutils web page:

http://www.python.org/sigs/distutils-sig/

The standalone version includes all of the code in this directory,
plus documentation, test scripts, examples, etc.

[Stephen_Sorkin · Accepted solution]

I tried this as well and encountered the same problem. We'll track it down and see the root cause, but there's an easy workaround that's preferable, especially if you're putting this search command into a separate app.

The active_directory python module appears to be just a single .py file. If you put this file in $SPLUNK_HOME/etc/apps/<app>/bin/, you should be able to import it from your search command. This technique should work with any pure python module (that doesn't require native C/C++ code to be compiled). You can see an example of this in the geoip app on splunkbase: http://www.splunkbase.com/apps/All/4.x/Add-On/app:Geo+Location+Lookup+Script. In that particular case a directory is placed in the app's bin directory.

[kholleran]

Thanks! I'll give that a try.