Developing for Splunk Enterprise

Can I add python modules to the Splunk environment?

Simeon
Splunk Employee
Splunk Employee

How can I add a python module that is not included in the Splunk python bundle? Specifically, I would like to use the pymssql module from within Splunk to run a scripted input.

Labels (1)
Tags (3)
1 Solution

amrit
Splunk Employee
Splunk Employee

There's also a more upgrade-friendly way to accomplish this. Some of our users setup whichever script they've configured in Splunk as a pass-through to a script that runs using their system Python (with whichever custom modules they've installed).

The steps are roughly:

  • configure your script in splunk (search script, scripted input, whatever)

  • this script should:

    • unset PYTHONPATH (in os.environ)

    • perhaps unset LD_LIBRARY_PATH, depending on your environment (also in os.environ)

    • create a process to run /usr/bin/python (via subprocess)

    • redirect stdin, stdout, stderr to/from script2

script2 can then load any arbitrary python module installed in your system's python installation.

View solution in original post

bhavikbhalodia
Path Finder

Just add below python code in your main script and it will work:

 

import os
import sys
import re

ta_name = '<splunk_app_name>'
ta_lib_name = '<parent_folder_of_python_pakage>'
pattern = re.compile(r"[\\/]etc[\\/]apps[\\/][^\\/]+[\\/]bin[\\/]?$")
new_paths = [path for path in sys.path if not pattern.search(path) or ta_name in path]
new_paths.insert(0, os.path.sep.join([ta_lib_name]))
sys.path = new_paths

 

 

inventsekar
Super Champion

Superb @bhavikbhalodia 

0 Karma

bhavikbhalodia
Path Finder

Thanks @inventsekar 

badarsebard
Communicator

I've recently released a new app that allows for exactly this kind of behavior. The PyDen app allows an administrator or developer to compile different versions of Python from source and create virtual environments. Once created, you can use the custom pip command from the search bar to install packages from PyPI. Once created, you can leverage the virtual environment by copying and importing an activation script provided with the app. When the custom command you create is run, it will use the virtual environment instead of the Splunk interpreter.

The app can be found here: https://splunkbase.splunk.com/app/4322/

BrunoLabelle
Explorer

Hey ebuitweb. if you put your library inside the splunk home... the minute you do an upgrade to Splunk... your libraries would get nuked... also if you have a cluster this would require you to login to every search head and edit the internal splunk home scripts directory... as well you may break the native product looking into that directory.... i would advise if you dont want headaches with support... dont put stuff in there... instead put it inside an app and deploy to the search heads. if you need more granular details about it, let me know and i can post a more granular detailed way to set it up and keep support on your side.

0 Karma

ebuitweb
Explorer

Hy,
a working workharound.

on your wrapper.sh, call your python script by using full path :

/usr/bin/python $SPLUNK_HOME/bin/scripts/yourpythonscript.py

0 Karma

BrunoLabelle
Explorer

although this would work technically... this would be a bad idea

Reasons:
if you add libraries in the native folder...
clustered environments would require you to login to every search heads to do this update.
you may by mistake break the native folder of splunk where it gets its own libraries.... Support wont be happy with you if they notice unwanted libraries in there.
when you upgrade Splunk software that directory gets nuked for the new release... meaning if you forget you may loose all your libraries.

To stay a happy admin and deploy quick and keep support on your side. you want to publish a hidden app with the libraries and reference them from there during execution of your code ... if your not familiar with this setup and want more details let me know and i can post a more detailed step to setting it up.

0 Karma

BrunoLabelle
Explorer

Best supported way to do this and still have Splunk support on your side.
Do not overwrite Splunk native libraries as you could break the product.
or even upgrades to the product or your server could break you.
What you do is import the library on the fly during execution of your script.

Your initial script would do these three lines at the top
import sys
sys.path.insert(0,'/Path of Library you want to add')
import YOURLIBRARY

Even better for clustered environments with many search heads out there.
Create an App on your cluster master responsible for deploying to the search heads and leave it hidden so it wont show on the searcheads.
call your app lets say
"prod_searchhead_bin"
in there you have the magic folder BIN where you put all you secret libraries.

then you command would look like this
import sys
sys.path.insert(0,'/opt/splunk/etc/apps/prod_searchhead_bin/bin')
import YOURLIBRARY

then when you publish to the cluster all search heads will comply and have libraries making it a breeze during upgrades/ updates.
if you upgrade or move environments. easy to deploy as it is a native app to deploy and make all your much needed libraries available.

0 Karma

joelurtubia
Explorer

Hello, I was the same problem with Mysql module that I was install on my Centos server

Splunk didn't work with this library, because splunk has they own python library...then you can fix it only added on the begin your script all libraries of python and also you must to add the python Centos library too... as this way

Find python packages

[root@xxxx]#find / -name site-packages
/usr/lib/python2.7/site-packages
/usr/lib64/python2.7/site-packages
/opt/splunk/etc/apps/Splunk_SA_Scientific_Python_linux_x86_64/bin/linux_x86_64/lib/python2.7/site-packages
/opt/splunk/lib/python2.7/site-packages

Find python binary

[root@xxxx]# whereis python
python: /usr/bin/python2.7 /usr/bin/python /usr/lib/python2.7 /usr/lib64/python2.7 /etc/python /usr/include/python2.7 /opt/splunk/bin/python /opt/splunk/bin/python2.7 /usr/share/man/man1/python.1.gz

include all at begin your script

import sys
sys.path.append('/usr/bin/python2.7')
sys.path.append('/usr/lib/python2.7/site-packages')
sys.path.append('/usr/lib64/python2.7/site-packages')

And that's it , you can run mysql module without any problem and create your alerts with this module.

Mysql Connection

import mysql.connector

I hope that this fix will help you
Joel Urtubia Ugarte

0 Karma

rlacher1
Explorer

virtualenv can be used to create a standalone Python installation with whatever modules are required. The scripted input could then use a wrapper script to activate the virtualenv, execute the Python script, and deactivate the virtualenv. This works okay if the native splunk module isn't required.

script.py

#!/usr/bin/env python
import package
package.do_stuff()

script_wrapper.sh

#!/bin/bash
source /path/to/virtual/env/bin/activate
python /path/to/script.py
deactivate

asifhj
Path Finder

Guys,

Its simple,
Copy your required packages/files from


/usr/local/lib/python2.7/dist-packages

to

/opt/splunk/lib/python2.7/site-packages/


hans
Splunk Employee
Splunk Employee

I just want to add how i installed ipython on splunk's python install. I downloaded the tar.gz and did:

$ splunk cmd python setup.py install

I got an error:

ImportError: No module named command.build_py

It turns out distutils in splunk's python install does not have the command module. The way I got around this is to use distutils from another python 2.7 install by adding its directory to PYTHONPATH:

$ export PYTHONPATH=$HOME/python272/lib/python2.7

and installed using setup.py again

$ splunk cmd python setup.py install

finally:

$ splunk cmd ipython
0 Karma

araitz
Splunk Employee
Splunk Employee

Here is an example, tested on Linux and Windows, that uses a non-splunk python but still lets you load most Splunk modules:

wrapper.py:

import os
import subprocess

_NEW_PYTHON_PATH = '/usr/bin/python'
_SPLUNK_PYTHON_PATH = os.environ['PYTHONPATH']

os.environ['PYTHONPATH'] = _NEW_PYTHON_PATH 
my_process = os.path.join(os.getcwd(), 'my_script.py')

p = subprocess.Popen([os.environ['PYTHONPATH'], my_process, _SPLUNK_PYTHON_PATH], 
                      stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
output = p.communicate()[0]
print output

my_script.py:

import sys 
import os

#This step is necessary in order to load splunk packages
from optparse import OptionParser
parser = OptionParser()
(options, args) = parser.parse_args()
_SPLUNK_PYTHON_PATH = args[0]
sys.path.append(_SPLUNK_PYTHON_PATH)

import splunk
import cherrypy
import some_package_from_new_python
...

jrouse025
Path Finder

Here is a full example @araitz thanks for the inspiration
link text

0 Karma

heshamzaid
Explorer

how to make this on django web framework
i make an splunk app and i need to import some non native python module

0 Karma

the_wolverine
Champion

Updating your Splunk Python is not supported (by Splunk). The recommended method is to update your system's Python or install a 3rd copy of Python that is not located in your PATH.

jrodman
Splunk Employee
Splunk Employee

No, there's no way to tell splunk to run some other python. However, you can make the script itself do this as per amrit's comment.

0 Karma

dianbo
New Member

i have 3 pythons in my linux server, local 2.3, 2.5 and 2.6(in splunk4.0.8). I can not install MySQL-python-1.2.3c1 to 2.6, but it's ok to install it to local 2.3. the following is the detail.

cd MySQL-python-1.2.3c1

python2.6 setup.py build ImportError: No module named setuptools

sh setuptools-0.6c11-py2.6.egg ImportError: No module named command.bdist

[root@localhost bin]# ./splunk cmd /local/dl/setuptools-0.6c11-py2.6.egg Traceback (most recent call last): File "", line 1, in File "/local/dl/setuptools-0.6c11-py2.6.egg/setuptools/init.py", line 2, in File "/local/dl/setuptools-0.6c11-py2.6.egg/setuptools/extension.py", line 2, in File "/local/dl/setuptools-0.6c11-py2.6.egg/setuptools/dist.py", line 5, in File "/local/dl/setuptools-0.6c11-py2.6.egg/setuptools/command/init.py", line 13, in ImportError: No module named command.bdist

[root@localhost bin]# ./splunk cmd /local/dl/MySQL_python-1.2.3c1-py2.6-linux-i686.egg couldn't run "/local/dl/MySQL_python-1.2.3c1-py2.6-linux-i686.egg": Exec format error

@the_wolverine, how to tell splunk to execute lookup python script with system's Python?

Thanks,

Dianbo

0 Karma

amrit
Splunk Employee
Splunk Employee

There's also a more upgrade-friendly way to accomplish this. Some of our users setup whichever script they've configured in Splunk as a pass-through to a script that runs using their system Python (with whichever custom modules they've installed).

The steps are roughly:

  • configure your script in splunk (search script, scripted input, whatever)

  • this script should:

    • unset PYTHONPATH (in os.environ)

    • perhaps unset LD_LIBRARY_PATH, depending on your environment (also in os.environ)

    • create a process to run /usr/bin/python (via subprocess)

    • redirect stdin, stdout, stderr to/from script2

script2 can then load any arbitrary python module installed in your system's python installation.

View solution in original post

Dark_Ichigo
Builder

Is there a step by step tutorial to accomplish this on a Linux environment?

Register for .conf21 Now! Go Vegas or Go Virtual!

How will you .conf21? You decide! Go in-person in Las Vegas, 10/18-10/21, or go online with .conf21 Virtual, 10/19-10/20.