Splunk Dev

Scripted Lookup Script doesn't work with Splunk Python version but works fine with Python 2.7

bansi
Path Finder

I have a Scripted Lookup Script which works great when ran with Python 2.7 version but fails under Splunk Python version 2.6 Here is the error

    c:\Splunk\etc\system\bin>splunk cmd python namelookup.py memberId memberName > memberInput.csv
Traceback (most recent call last):
  File "namelookup.py", line 7, in <module>
    import pyodbc
ImportError: DLL load failed: The specified module could not be found.

So from the error its quite understandable, the pyodbc, the python extension module needs to be run under Splunk environment

But Splunk Python version 2.6 doesnt support pyodbc module on 64bit OS hence i have to install Python 2.7 with pyodbc module underneath it and the below scripted lookup script works fine

C:\Python27\python.exe namelookup.py memberId memberName < memberInput.csv

So how would i make the same scripted lookup script work under Splunk environment i.e.

 splunk cmd python namelookup.py memberId memberName > memberInput.csv

I have made the following changes but no help

  1. import os del os.environ[PYTHONPATH] del os.environ[LD_LIBRARY_PATH]

    ppath = "C:\Python27" spath = "C:\Splunk\etc\system\bin\namelookup.py"

    os.execv(ppath, [spath] + os.argv[1:])

  2. i have copied

    C:\Python27\lib\site-packages, TO $SPLUNK_HOME\lib\Python2.6\site-packages

but still it doesnt help

0 Karma

asifhj
Path Finder

Guys,

Its simple,
Copy your required packages/files from


/usr/local/lib/python2.7/dist-packages

to

/opt/splunk/lib/python2.7/site-packages/


0 Karma

bansi
Path Finder

LookUp Script NOT To Read from CSV File Yes i am looking for script which reads from the context of Splunk Searching instead of taking the input from CSV file from stdin

Here is the script which works from standalone but not from context of splunk search

  # File namelookup.py
# ------------------------------
import os,csv
import sys
import logging
import logging.config
import pyodbc

FIELDS = [ "memberId", "memberName" ]
# Given a id, find its name
def lookupName(idf, cur):
        try:
        selString = "SELECT first_name FROM emp where Member_ID="
        cur.execute (selString + idf )
        row = cur.fetchone()
        return row[0]
    except:
        return []





def main():

       if len(sys.argv) != 3:
          print "Usage: python name_lookup.py [id field] [name field]"
          sys.exit(0)

       idf = sys.argv[1]
       namef = sys.argv[2]
       #THIS IS NOT DESIRED: r = csv.reader(sys.stdin)
       #r = csv.reader(open('C:\\Splunk\\etc\\system\\bin\\memberInput.csv'))
       #r = csv.reader(sys.stdin)
       #w = csv.DictWriter(sys.stdout, FEILDS)
       result = {}
       w = csv.DictWriter(sys.stdout, FIELDS)
       header = []
       first = True
       conn = pyodbc.connect('DSN=DBT1')
       cursor = conn.cursor()

       if len( idf):
            result[namef] = lookupName(idf, cursor)
            print("Result of"+ namef)
            print(result[namef])
            if len(result[namef]):
                w.writerow(result)



       cursor.close()
       conn.close()         

main()

Pl let me know how to rewrite the script to read/write from the context of splunk search

0 Karma

bansi
Path Finder

Thanks once again for quick response. I greatly appreciate it."The context of Splunk Search" means from Splunk Web the search command "source="Test_Log.txt" | xmlkv entry | lookup namelookupWrapper memberId | table memberId, memberName" doesn't return any data under "memberName" column. So i am not sure how to check if Splunk Web successfully invokes namelookupWrapper script which you suggested. Also the real script i.e. namelookup.py has code snippet "r = csv.reader(sys.stdin)" so wondering how would namelookWrapper.py would re-direct sys.stdin to real script namelookup.py.

0 Karma

Lowell
Super Champion

Or, if you want to do something with the final results of your command, then you can look at running an alert script. See http://answers.splunk.com/questions/tagged/alert-scripts

0 Karma

Lowell
Super Champion

Please clarify what you mean by "reads from the context of Splunk Searching". I don't know what you are asking. If you are looking to gain access to the events/results of your search, then you want to make a custom search command rather than a lookup. See: http://answers.splunk.com/questions/tagged/custom-search-script

0 Karma

Lowell
Super Champion

Just a security note: Please be aware that your current code is open to a sql-injection type of attack. For example, if one of your events contained a malicious "memberId" field with the literal value: "5; DROP TABLE emp;" you could end up allowing very bad things to happen to your database. This is easily avoidable, simply use a parameter-based query instead of doing string concatenation (i.e. selString + idf). You should be able to find some examples of how to do this in the pyodbc docs. BTW, using parameters also allows additional SQL optimizations.

0 Karma

Lowell
Super Champion

Because an external lookup must use the built-in python interpreter, it looks like you'll have to make a simple wrapper python script that runs an external command, which just also happens to be another python executable.

namedlookup_wrapper.py:

import os, sys

# Remove problematic environmental variables if they exist.
for envvar in ("PYTHONPATH", "LD_LIBRARY_PATH"):
    if envvar in os.environ:
        del os.environ[envvar]

python_executable = r"C:\Python27\python.exe"
# Attempt to figure out the location of the real script; or you can just hard-code the full path here.  (I think this works, but I didn't double check it.
real_script = os.path.join(os.path.dirname(__file__), "namedlookup.py")

os.execv(python_executable, [ python_executable, real_script ] + sys.argv[1:])

Then replace "namedlookup.py" with "namedlookup_wrapper.py" in your lookup definition.

FYI. Here is a similar issue with scripted inputs:

http://answers.splunk.com/questions/2821/python-scripted-inputs-run-with-the-wrong-version-of-python...


Alternately, you could use a different python database driver. If you're connecting to SQL Server, I'd suggested pymssql. If you really need true ODBC support, you could look into the commercial mxODBC driver (but that will cost you). I've done this for a number of database scripted inputs that I use.

But yeah, be careful messing around too much with Splunk's build in site-packages folder. Instead I recommend creating .egg files, putting them in your own lib folder, and explicitly doing a sys.path.append("/your/lib/whatever.egg") before your import. But so far any approach I've found to extend the built in python environment has been a little bit delicate to maintain. Of course, the wrapper approach is quite easy, but it has the implicit overhead of calling an extra python interpreter; so you have to choose what's most important to you.

stefano_guidoba
Communicator

works great! Thanks!

0 Karma

bansi
Path Finder

Thanks Lowell. You made my day and definately deserve credit for the answer. I would greatly appreciate your help in re-writing the script to read from the context of splunk search instead of CSV file from stdin. I am posting it in the below section

0 Karma

Lowell
Super Champion

Whoops, that example python code was rather buggy. I've updated it, and I'll go back and fix it in the places where I copied it from. Please let me know if the "__file__" thing works for scripted lookups. I think it should work, but I'd like confirmation. (That trick assumes that both scripts are in the same directory, which is normally going to be true.)

0 Karma

bansi
Path Finder

I am still getting errors:
splunk cmd python namelookupWrapper.py

Error File "C:\Splunk\Python-2.6\Lib\os.py", line 432, in delitem
del self.data[key.upper()]
KeyError: 'LD_LIBRARY_PATH'
So i commented that line still error persist in next line
splunk cmd python namelookupWrapper.py memberId memberName < memberInput.csv
File "c:\Splunk\etc\system\bin\namelookupWrapper.py", line 7, in
os.execv("C:\Python27\python.exe", [ "C:\Splunk\etc\system\bin\namelookup.py" ] + os.argv[1:])
AttributeError: 'module' object has no attribute 'argv'

0 Karma
Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

Splunk is officially part of Cisco

Revolutionizing how our customers build resilience across their entire digital footprint.   Splunk ...

Splunk APM & RUM | Planned Maintenance March 26 - March 28, 2024

There will be planned maintenance for Splunk APM and RUM between March 26, 2024 and March 28, 2024 as ...