Wondering if Splunk latest version support Python 2.7 otherwise user community has compatability issues ... View more

I have following two python scripts -namelookupWrapper.py -namelookup.py The namelookupWrapper.py takes input of "memberId", "memberName" from Splunk Web interface and has following code snippet idf = sys.argv[1] namef = sys.argv[2] real_script = "C:\\Splunk\\etc\\apps\\search\\bin\\namelookup.py" r = csv.reader(sys.stdin) os.execv(python_executable, [ python_executable, real_script ] + sys.argv[1:] ) Wondering how would i pass csv reader object "r" as an argument using os.execv() to another python script i.e. namelookup.py Using the csv reader object i iterate thru the input in namelookupWrapper.py which looks like as shown below [MemberId, MemberName] [123, ] [456, ] [989, ] Now i have another script i.e. namelookup.py running under Python 2.7 using pyodbc to retrieve Member Names from database for a given Member Id in namelooupWrapper.py Please note the reason i need to do so is Splunk Python 2.6 host namelookupWrapper.py and the real_script namelookup.py is under Python 2.7 has pyodbc connection ... View more

From the url http://blogs.splunk.com/2009/09/14/enriching-data-with-db-lookups-part-2/ i read the following excerpt “The Python program gets its city field input via standard CSV input from Splunk, calls SQL to find the corresponding country, and produces the aggregate CSV output that contains the city with its correlated country” On further analyzing this statement I infer “Splunk Web interface Serializes input to Stdin and the Python Script using a CSV reader object reads the input from Stdin” Hence the presence of following code snippet in Python external lookup script r = csv.reader(sys.stdin) Having said that the value of variable “r” is empty and the lookup script does nothing I arrived at this statement by printing debugging statements to log file Is there anyone who can help with this? ... View more

I have a Scripted Lookup Script which works great when ran with Python 2.7 version but fails under Splunk Python version 2.6 Here is the error c:\Splunk\etc\system\bin>splunk cmd python namelookup.py memberId memberName > memberInput.csv Traceback (most recent call last): File "namelookup.py", line 7, in <module> import pyodbc ImportError: DLL load failed: The specified module could not be found. So from the error its quite understandable, the pyodbc, the python extension module needs to be run under Splunk environment But Splunk Python version 2.6 doesnt support pyodbc module on 64bit OS hence i have to install Python 2.7 with pyodbc module underneath it and the below scripted lookup script works fine C:\Python27\python.exe namelookup.py memberId memberName < memberInput.csv So how would i make the same scripted lookup script work under Splunk environment i.e. splunk cmd python namelookup.py memberId memberName > memberInput.csv I have made the following changes but no help import os del os.environ[PYTHONPATH] del os.environ[LD_LIBRARY_PATH] ppath = "C:\Python27" spath = "C:\Splunk\etc\system\bin\namelookup.py" os.execv(ppath, [spath] + os.argv[1:]) i have copied C:\Python27\lib\site-packages, TO $SPLUNK_HOME\lib\Python2.6\site-packages but still it doesnt help ... View more

Below is the props.conf at $SPLUNK_HOME/etc/system/local: [Test_Log] lookup_table = namelookup memberId OUTPUT memberName Below is the transforms.conf at $SPLUNK_HOME/etc/system/local: [namelookup] external_cmd = namelookup.py memberId memberName external_type = python fields_list = memberId, memberName Script location : $SPLUNK_HOME/etc/system/bin/namelookup.py # File namelookup.py # ------------------------------ import os,csv import sys import logging import logging.config import pyodbc # Given a id, find its name def lookupName(idf, cur): #logger.debug("====Inside Lookup idf====="+idf) try: selString = "SELECT first_name FROM emp where Member_ID=" cur.execute (selString + idf ) row = cur.fetchone() return row[0] except: return [] def main(): logging.config.fileConfig("logging.conf") logger = logging.getLogger("name_lookup") if len(sys.argv) != 3: print "Usage: python name_lookup.py [id field] [name field]" sys.exit(0) idf = sys.argv[1] namef = sys.argv[2] #logger.debug("====idf===="+idf) #logger.debug("====namef====="+namef) r = csv.reader(sys.stdin) w = None header = [] first = True conn = pyodbc.connect('DSN=IBM_DB') cursor = conn.cursor() for line in r: if first: header = line if idf not in header or namef not in header: print "Host and IP fields must exist in CSV data" sys.exit(0) csv.writer(sys.stdout).writerow(header) w = csv.DictWriter(sys.stdout, header) first = False continue # Read the result result = {} i = 0 while i < len(header): if i < len(line): result[header[i]] = line[i] else: result[header[i]] = '' i += 1 # Perform the lookup for city to country if necessary if len(result[idf]) and len(result[namef]): w.writerow(result) elif len(result[idf]): result[namef] = lookupName(result[idf], cursor) if len(result[namef]): w.writerow(result) cursor.close() conn.close() main() the results displayed on stdout are as desired C:\Splunk\etc\system\bin>db_lookup.py memberId memberName < memberInput.csv produces following output retrieving memberName from database memberId,memberName 006,RANDY 007,LEONY 009,RANDOLPH However i still have following issues The script doesn't output similar results in the CSV file. Wondering why it doesn't output in CSV file when the desired results are displayed in STDOUT. So what i am missing here? The script then outputs from the CSV file and returns it to Splunk, which populates the memberName field in your results source="Test_Log.txt" | xmlkv entry | lookup namelookup memberId OUTPUT memberName | table memberId, memberName Please let me know how to populate CSV file with the results ... View more

We have a requirement to send secure emails with encryption something like PGP Encrypt/Decrypt email feature so wondering if Splunk "sendmail" utility is capable of sending secure emails ... View more

Below is the props.conf at $SPLUNK_HOME/etc/system/local: [SPLUNK_SERVICE_Log] lookup_table = namelookup Id OUTPUT Name Below is the transforms.conf at $SPLUNK_HOME/etc/system/local: [namelookup] external_cmd = namelookup.py Id Name external_type = python fields_list = Id, Name Script location : $SPLUNK_HOME/etc/system/bin/namelookup.py # File namelookup.py # ------------------------------ import os,csv #import pyodbc import sys import logging import logging.config def main(): #if len(sys.argv) != 3: #print "Usage: python name_lookup.py [id field] [name field]" #sys.exit(0) logging.config.fileConfig("logging.conf") # create logger logger = logging.getLogger("namelookup") # "application" code logger.debug("====Inside Main=====") idf = sys.argv[1] namef = sys.argv[2] r = csv.reader(sys.stdin) w = None header = [] first = True d1 = {} # Add items d1["006981166"] = "John" d1["007094117"] = "Mike" d1["007094118"] = "Scott" for line in r: if first: header = line print "Header:", header if idf not in header or namef not in header: print "Id and Name fields must exist in CSV data" sys.exit(0) csv.writer(sys.stdout).writerow(header) w = csv.DictWriter(sys.stdout, header) first = False continue # Read the result result = {} i = 0 while i < len(header): if i < len(line): result[header[i]] = line[i] else: result[header[i]] = '' i += 1 # Perform the lookup if len(result[idf]) and len(result[namef]) : w.writerow(result) elif len(result[idf]): result[namef] = lookup(result[idf], d1) if len(result[namef]): w.writerow(result) # Given a Id, find its Name def lookup(id, d1): try: for key in d1.keys(): if key == id: #print "Value=", d1[key] return d1[key] except: return [] main() However, when I run the below search, It doesn't return any search results under name source="Test_Log.txt" | xmlkv entry | lookup namelookup Id OUTPUT Name | table Id, name Please let me know where i am going wrong in the script or where exactly is the script failing. Is their a way to debug the script using Komodo Edit IDE . I want debugger to launch the moment you hit enter in the Splunk Web Interface because i am not even sure the script is invoked by Splunk. So i would like to see atleast the first print statement in the script is printed onto console. When i tried to run as standlone program using the command splunk cmd namelookup.py 123 it opens a command prompt and immediately closes it. So not sure whats going on with this script ... View more