topic Re: Scripted Lookup Script doesn't work with Splunk Python version but works fine with Python 2.7 in Splunk Dev

Scripted Lookup Script doesn't work with Splunk Python version but works fine with Python 2.7

bansi — Fri, 21 Jan 2011 04:03:16 GMT

I have a Scripted Lookup Script which works great when ran with Python 2.7 version but fails under Splunk Python version 2.6 Here is the error

    c:\Splunk\etc\system\bin>splunk cmd python namelookup.py memberId memberName > memberInput.csv
Traceback (most recent call last):
  File "namelookup.py", line 7, in <module>
    import pyodbc
ImportError: DLL load failed: The specified module could not be found.

So from the error its quite understandable, the pyodbc, the python extension module needs to be run under Splunk environment

But Splunk Python version 2.6 doesnt support pyodbc module on 64bit OS hence i have to install Python 2.7 with pyodbc module underneath it and the below scripted lookup script works fine

C:\Python27\python.exe namelookup.py memberId memberName < memberInput.csv

So how would i make the same scripted lookup script work under Splunk environment i.e.

 splunk cmd python namelookup.py memberId memberName > memberInput.csv

I have made the following changes but no help

import os del os.environ[PYTHONPATH] del os.environ[LD_LIBRARY_PATH]

ppath = "C:\Python27" spath = "C:\Splunk\etc\system\bin\namelookup.py"

os.execv(ppath, [spath] + os.argv[1:])
i have copied

C:\Python27\lib\site-packages, TO $SPLUNK_HOME\lib\Python2.6\site-packages

but still it doesnt help

Re: Scripted Lookup Script doesn't work with Splunk Python version but works fine with Python 2.7

Lowell — Fri, 21 Jan 2011 05:10:50 GMT

Because an external lookup must use the built-in python interpreter, it looks like you'll have to make a simple wrapper python script that runs an external command, which just also happens to be another python executable.

namedlookup_wrapper.py:

import os, sys

# Remove problematic environmental variables if they exist.
for envvar in ("PYTHONPATH", "LD_LIBRARY_PATH"):
    if envvar in os.environ:
        del os.environ[envvar]

python_executable = r"C:\Python27\python.exe"
# Attempt to figure out the location of the real script; or you can just hard-code the full path here.  (I think this works, but I didn't double check it.
real_script = os.path.join(os.path.dirname(__file__), "namedlookup.py")

os.execv(python_executable, [ python_executable, real_script ] + sys.argv[1:])

Then replace "namedlookup.py" with "namedlookup_wrapper.py" in your lookup definition.

FYI. Here is a similar issue with scripted inputs:

http://answers.splunk.com/questions/2821/python-scripted-inputs-run-with-the-wrong-version-of-python-in-splunk-4-1-2

Alternately, you could use a different python database driver. If you're connecting to SQL Server, I'd suggested pymssql. If you really need true ODBC support, you could look into the commercial mxODBC driver (but that will cost you). I've done this for a number of database scripted inputs that I use.

But yeah, be careful messing around too much with Splunk's build in site-packages folder. Instead I recommend creating .egg files, putting them in your own lib folder, and explicitly doing a sys.path.append("/your/lib/whatever.egg") before your import. But so far any approach I've found to extend the built in python environment has been a little bit delicate to maintain. Of course, the wrapper approach is quite easy, but it has the implicit overhead of calling an extra python interpreter; so you have to choose what's most important to you.

Re: Scripted Lookup Script doesn't work with Splunk Python version but works fine with Python 2.7

bansi — Mon, 28 Sep 2020 09:23:30 GMT

I am still getting errors:
splunk cmd python namelookupWrapper.py

Error File "C:\Splunk\Python-2.6\Lib\os.py", line 432, in delitem
del self.data[key.upper()]
KeyError: 'LD_LIBRARY_PATH'
So i commented that line still error persist in next line
splunk cmd python namelookupWrapper.py memberId memberName < memberInput.csv
File "c:\Splunk\etc\system\bin\namelookupWrapper.py", line 7, in
os.execv("C:\Python27\python.exe", [ "C:\Splunk\etc\system\bin\namelookup.py" ] + os.argv[1:])
AttributeError: 'module' object has no attribute 'argv'

Re: Scripted Lookup Script doesn't work with Splunk Python version but works fine with Python 2.7

Lowell — Fri, 21 Jan 2011 23:09:08 GMT

Whoops, that example python code was rather buggy. I've updated it, and I'll go back and fix it in the places where I copied it from. Please let me know if the "__file__" thing works for scripted lookups. I think it should work, but I'd like confirmation. (That trick assumes that both scripts are in the same directory, which is normally going to be true.)

Re: Scripted Lookup Script doesn't work with Splunk Python version but works fine with Python 2.7

bansi — Mon, 24 Jan 2011 10:19:56 GMT

Thanks Lowell. You made my day and definately deserve credit for the answer. I would greatly appreciate your help in re-writing the script to read from the context of splunk search instead of CSV file from stdin. I am posting it in the below section

Re: Scripted Lookup Script doesn't work with Splunk Python version but works fine with Python 2.7

bansi — Mon, 24 Jan 2011 10:24:27 GMT

LookUp Script NOT To Read from CSV File Yes i am looking for script which reads from the context of Splunk Searching instead of taking the input from CSV file from stdin

Here is the script which works from standalone but not from context of splunk search

  # File namelookup.py
# ------------------------------
import os,csv
import sys
import logging
import logging.config
import pyodbc

FIELDS = [ "memberId", "memberName" ]
# Given a id, find its name
def lookupName(idf, cur):
        try:
        selString = "SELECT first_name FROM emp where Member_ID="
        cur.execute (selString + idf )
        row = cur.fetchone()
        return row[0]
    except:
        return []





def main():

       if len(sys.argv) != 3:
          print "Usage: python name_lookup.py [id field] [name field]"
          sys.exit(0)

       idf = sys.argv[1]
       namef = sys.argv[2]
       #THIS IS NOT DESIRED: r = csv.reader(sys.stdin)
       #r = csv.reader(open('C:\\Splunk\\etc\\system\\bin\\memberInput.csv'))
       #r = csv.reader(sys.stdin)
       #w = csv.DictWriter(sys.stdout, FEILDS)
       result = {}
       w = csv.DictWriter(sys.stdout, FIELDS)
       header = []
       first = True
       conn = pyodbc.connect('DSN=DBT1')
       cursor = conn.cursor()

       if len( idf):
            result[namef] = lookupName(idf, cursor)
            print("Result of"+ namef)
            print(result[namef])
            if len(result[namef]):
                w.writerow(result)



       cursor.close()
       conn.close()         

main()

Pl let me know how to rewrite the script to read/write from the context of splunk search

Re: Scripted Lookup Script doesn't work with Splunk Python version but works fine with Python 2.7

Lowell — Tue, 25 Jan 2011 01:18:11 GMT

Just a security note: Please be aware that your current code is open to a sql-injection type of attack. For example, if one of your events contained a malicious "memberId" field with the literal value: "5; DROP TABLE emp;" you could end up allowing very bad things to happen to your database. This is easily avoidable, simply use a parameter-based query instead of doing string concatenation (i.e. selString + idf). You should be able to find some examples of how to do this in the pyodbc docs. BTW, using parameters also allows additional SQL optimizations.

Re: Scripted Lookup Script doesn't work with Splunk Python version but works fine with Python 2.7

Lowell — Tue, 25 Jan 2011 01:20:35 GMT

Please clarify what you mean by "reads from the context of Splunk Searching". I don't know what you are asking. If you are looking to gain access to the events/results of your search, then you want to make a custom search command rather than a lookup. See: http://answers.splunk.com/questions/tagged/custom-search-script

Re: Scripted Lookup Script doesn't work with Splunk Python version but works fine with Python 2.7

Lowell — Tue, 25 Jan 2011 01:30:51 GMT

Or, if you want to do something with the final results of your command, then you can look at running an alert script. See http://answers.splunk.com/questions/tagged/alert-scripts

Re: Scripted Lookup Script doesn't work with Splunk Python version but works fine with Python 2.7

bansi — Tue, 25 Jan 2011 01:57:52 GMT

Thanks once again for quick response. I greatly appreciate it."The context of Splunk Search" means from Splunk Web the search command "source="Test_Log.txt" | xmlkv entry | lookup namelookupWrapper memberId | table memberId, memberName" doesn't return any data under "memberName" column. So i am not sure how to check if Splunk Web successfully invokes namelookupWrapper script which you suggested. Also the real script i.e. namelookup.py has code snippet "r = csv.reader(sys.stdin)" so wondering how would namelookWrapper.py would re-direct sys.stdin to real script namelookup.py.

Re: Scripted Lookup Script doesn't work with Splunk Python version but works fine with Python 2.7

asifhj — Fri, 27 Sep 2013 14:09:52 GMT

Guys,

Its simple,
Copy your required packages/files from

/usr/local/lib/python2.7/dist-packages

to

/opt/splunk/lib/python2.7/site-packages/

Re: Scripted Lookup Script doesn't work with Splunk Python version but works fine with Python 2.7

stefano_guidoba — Thu, 10 Oct 2013 10:35:38 GMT

works great! Thanks!