How to use a different Python version with subprocess to create search script using custom command?

jrouse025 — Thu, 05 Jan 2023 20:20:33 GMT

Purpose: Help the reader understand one approach to have Splunk kick-off a Python script that returns data to Splunk without indexing returned data, or modifying the Slunk Python installation.

Use Case: Splunk ships with a minimal implementation of Python 2.7. If you need to utilize a module that isn’t installed with the Splunk implementation it is recommended that an installation external to Splunk, with the modules installed, be used.

Expected Action: The user will initiate a Splunk search query using a custom command. This custom command will initiate a wrapper python script that will read a passed parameter/argument and initiate the target python script and return the values to the Slunk interface for display or use.

(Note: this is Splunk Enterprise on Windows Server)

Python Needs:

Python Installation (external to Splunk Python installation) -Full Python or Virtual Environment.
Install your required modules into your python environment (pip or other installation tool)
Note the Python executable locations -Splunk Python location: (e.g. D:\Splunk\bin\python.exe) -External Python installation (e.g. D:\Splunk_Python_venv\p27_15\Scripts\python.exe)

Script Requirements:
1. Wrapper script: (see example_wrapper.py)
-Will work with Splunk python implementation. (No modules added outside of Splunk standard installation)
-Reference Intersplunk class
sys.path.append("/Splunk/Python-2.7/Lib/site-packages/splunk")
import Intersplunk
-Print output results from target script
2. Target script: (see example_target.py)
-Can use your modules or customizations
-Reference Intersplunk class
sys.path.append("/Splunk/Python-2.7/Lib/site-packages/splunk")
import Intersplunk
-Call Intersplunk.outputResults(results)
(results) is your result set and it must be in a format that can be consumed by Splunk.
Splunk Custom Search Command:
1. Create custom search command in commands.conf
[activatepython]
chuncked = false
filename = example_wrapper.py

Run Script from commandline:
\Splunk\etc\apps\myapp\bin>........\bin\splunk cmd python example_wrapper.py scripttarget=example_target.py

Run Script from SPL

Example Files:

######################################################################################
############### following is the example_wrapper.py file #############################
######################################################################################
import os
import subprocess
import re
import sys
sys.path.append("/Splunk/Python-2.7/Lib/site-packages/splunk")
import Intersplunk


custompythonpath = 'D:\Splunk_Python_venv\p27_15\Scripts\python.exe'
splunkpythonpath = 'D:\Splunk\bin\python.exe'

os.environ['PYTHONPATH'] = custompythonpath
# from splunk web interface this is needed to get the arguments
# example SPL "| scriptwrapper scripttarget=example_target.py"
keywords, argvals = Intersplunk.getKeywordsAndOptions()
try:
    #pull in the arguments
    scripttarget = argvals['scripttarget']
    #prevent code injection: verify we aren't getting a file outside of our run directory 
    fileVerify=re.split('/|\*|\\\\|\'|\"', scripttarget)
    if os.path.isfile(fileVerify[0]):
        #create full path to your script
        my_process = os.path.join(os.getcwd(), scripttarget)
        #create subprocess be sure to pass stdin, stdout, stderr
        p = subprocess.Popen([custompythonpath, my_process, splunkpythonpath], stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
        # get the output
        output = p.communicate()[0]
        # push it out
        print output
    else:
        print ('Unable to verify requested action'.format(fileVerify))
except Exception as e:
    print(e)


######################################################################################
############### following is the example_target.py file #############################
######################################################################################


# pypyodbc has a dependency on environment variable 'windir' be sure 
# if using the the activate process that the class passes in the evirnoment variable on activate.
import sys
import pypyodbc
import sys
import os
sys.path.append("/Splunk/Python-2.7/Lib/site-packages/splunk")
import Intersplunk

def GetSqlResults(connectionString, sqlText):
    results = []
    try:
        db = pypyodbc.connect(connectionString)
        cursor = db.cursor()
        cursor.execute(sqlText)
        columns = [column[0] for column in cursor.description]
        #had to do the following to get splunk to accept the result set (I thought this would be simpler)
        for row in cursor.fetchall():
            result = {}
            for column in columns:
                result[column] = row[column]
            results.append(result)
        db.close()
    except Exception as e:
        print('Bad things happened when executing SQL: {0}'.format(e))

    return results


if __name__ == "__main__":
    connectionString = "Trusted_Connection=yes; Driver={SQL Server};server=greatstuffserver;database=greatstuffdatabase'
    sqlText = 'select id, name from greatstufftable'
    results = []
    results = GetSqlResults(connectionString, sqlText)
    #send results to Intersplunk (the wrapper class will 'print' the output)
    Intersplunk.outputResults(results)
    sys.exit(0)

Re: Use a different Python version with subprocess to create search script using custom command?

spunk_enthusias — Thu, 05 Jan 2023 20:16:22 GMT

Thank you! This may come in handy if I don't manage to find another option to keep the search command's process alive and end up having to manage one myself.

topic Re: Use a different Python version with subprocess to create search script using custom command? in Splunk Dev

How to use a different Python version with subprocess to create search script using custom command?

Re: Use a different Python version with subprocess to create search script using custom command?