Developing for Splunk Enterprise

Scripted Lookup Script doesn't work with Splunk Python version but works fine with Python 2.7

Path Finder

I have a Scripted Lookup Script which works great when ran with Python 2.7 version but fails under Splunk Python version 2.6 Here is the error

    c:\Splunk\etc\system\bin>splunk cmd python memberId memberName > memberInput.csv
Traceback (most recent call last):
  File "", line 7, in <module>
    import pyodbc
ImportError: DLL load failed: The specified module could not be found.

So from the error its quite understandable, the pyodbc, the python extension module needs to be run under Splunk environment

But Splunk Python version 2.6 doesnt support pyodbc module on 64bit OS hence i have to install Python 2.7 with pyodbc module underneath it and the below scripted lookup script works fine

C:\Python27\python.exe memberId memberName < memberInput.csv

So how would i make the same scripted lookup script work under Splunk environment i.e.

 splunk cmd python memberId memberName > memberInput.csv

I have made the following changes but no help

  1. import os del os.environ[PYTHONPATH] del os.environ[LD_LIBRARY_PATH]

    ppath = "C:\Python27" spath = "C:\Splunk\etc\system\bin\"

    os.execv(ppath, [spath] + os.argv[1:])

  2. i have copied

    C:\Python27\lib\site-packages, TO $SPLUNK_HOME\lib\Python2.6\site-packages

but still it doesnt help

0 Karma

Path Finder


Its simple,
Copy your required packages/files from




0 Karma

Path Finder

LookUp Script NOT To Read from CSV File Yes i am looking for script which reads from the context of Splunk Searching instead of taking the input from CSV file from stdin

Here is the script which works from standalone but not from context of splunk search

  # File
# ------------------------------
import os,csv
import sys
import logging
import logging.config
import pyodbc

FIELDS = [ "memberId", "memberName" ]
# Given a id, find its name
def lookupName(idf, cur):
        selString = "SELECT first_name FROM emp where Member_ID="
        cur.execute (selString + idf )
        row = cur.fetchone()
        return row[0]
        return []

def main():

       if len(sys.argv) != 3:
          print "Usage: python [id field] [name field]"

       idf = sys.argv[1]
       namef = sys.argv[2]
       #THIS IS NOT DESIRED: r = csv.reader(sys.stdin)
       #r = csv.reader(open('C:\\Splunk\\etc\\system\\bin\\memberInput.csv'))
       #r = csv.reader(sys.stdin)
       #w = csv.DictWriter(sys.stdout, FEILDS)
       result = {}
       w = csv.DictWriter(sys.stdout, FIELDS)
       header = []
       first = True
       conn = pyodbc.connect('DSN=DBT1')
       cursor = conn.cursor()

       if len( idf):
            result[namef] = lookupName(idf, cursor)
            print("Result of"+ namef)
            if len(result[namef]):



Pl let me know how to rewrite the script to read/write from the context of splunk search

0 Karma

Path Finder

Thanks once again for quick response. I greatly appreciate it."The context of Splunk Search" means from Splunk Web the search command "source="Test_Log.txt" | xmlkv entry | lookup namelookupWrapper memberId | table memberId, memberName" doesn't return any data under "memberName" column. So i am not sure how to check if Splunk Web successfully invokes namelookupWrapper script which you suggested. Also the real script i.e. has code snippet "r = csv.reader(sys.stdin)" so wondering how would would re-direct sys.stdin to real script

0 Karma

Super Champion

Or, if you want to do something with the final results of your command, then you can look at running an alert script. See

0 Karma

Super Champion

Please clarify what you mean by "reads from the context of Splunk Searching". I don't know what you are asking. If you are looking to gain access to the events/results of your search, then you want to make a custom search command rather than a lookup. See:

0 Karma

Super Champion

Just a security note: Please be aware that your current code is open to a sql-injection type of attack. For example, if one of your events contained a malicious "memberId" field with the literal value: "5; DROP TABLE emp;" you could end up allowing very bad things to happen to your database. This is easily avoidable, simply use a parameter-based query instead of doing string concatenation (i.e. selString + idf). You should be able to find some examples of how to do this in the pyodbc docs. BTW, using parameters also allows additional SQL optimizations.

0 Karma

Super Champion

Because an external lookup must use the built-in python interpreter, it looks like you'll have to make a simple wrapper python script that runs an external command, which just also happens to be another python executable.

import os, sys

# Remove problematic environmental variables if they exist.
for envvar in ("PYTHONPATH", "LD_LIBRARY_PATH"):
    if envvar in os.environ:
        del os.environ[envvar]

python_executable = r"C:\Python27\python.exe"
# Attempt to figure out the location of the real script; or you can just hard-code the full path here.  (I think this works, but I didn't double check it.
real_script = os.path.join(os.path.dirname(__file__), "")

os.execv(python_executable, [ python_executable, real_script ] + sys.argv[1:])

Then replace "" with "" in your lookup definition.

FYI. Here is a similar issue with scripted inputs:

Alternately, you could use a different python database driver. If you're connecting to SQL Server, I'd suggested pymssql. If you really need true ODBC support, you could look into the commercial mxODBC driver (but that will cost you). I've done this for a number of database scripted inputs that I use.

But yeah, be careful messing around too much with Splunk's build in site-packages folder. Instead I recommend creating .egg files, putting them in your own lib folder, and explicitly doing a sys.path.append("/your/lib/whatever.egg") before your import. But so far any approach I've found to extend the built in python environment has been a little bit delicate to maintain. Of course, the wrapper approach is quite easy, but it has the implicit overhead of calling an extra python interpreter; so you have to choose what's most important to you.


works great! Thanks!

0 Karma

Path Finder

Thanks Lowell. You made my day and definately deserve credit for the answer. I would greatly appreciate your help in re-writing the script to read from the context of splunk search instead of CSV file from stdin. I am posting it in the below section

0 Karma

Super Champion

Whoops, that example python code was rather buggy. I've updated it, and I'll go back and fix it in the places where I copied it from. Please let me know if the "__file__" thing works for scripted lookups. I think it should work, but I'd like confirmation. (That trick assumes that both scripts are in the same directory, which is normally going to be true.)

0 Karma

Path Finder

I am still getting errors:
splunk cmd python

Error File "C:\Splunk\Python-2.6\Lib\", line 432, in delitem
So i commented that line still error persist in next line
splunk cmd python memberId memberName < memberInput.csv
File "c:\Splunk\etc\system\bin\", line 7, in
os.execv("C:\Python27\python.exe", [ "C:\Splunk\etc\system\bin\" ] + os.argv[1:])
AttributeError: 'module' object has no attribute 'argv'

0 Karma
*NEW* Splunk Love Promo!
Snag a $25 Visa Gift Card for Giving Your Review!

It's another Splunk Love Special! For a limited time, you can review one of our select Splunk products through Gartner Peer Insights and receive a $25 Visa gift card!


Or Learn More in Our Blog >>