Splunk Dev

How to use a different Python version with subprocess to create search script using custom command?

jrouse025
Path Finder

Purpose: Help the reader understand one approach to have Splunk kick-off a Python script that returns data to Splunk without indexing returned data, or modifying the Slunk Python installation.

Use Case: Splunk ships with a minimal implementation of Python 2.7. If you need to utilize a module that isn’t installed with the Splunk implementation it is recommended that an installation external to Splunk, with the modules installed, be used.

Expected Action: The user will initiate a Splunk search query using a custom command. This custom command will initiate a wrapper python script that will read a passed parameter/argument and initiate the target python script and return the values to the Slunk interface for display or use.

(Note: this is Splunk Enterprise on Windows Server)

Python Needs:

  1. Python Installation (external to Splunk Python installation) -Full Python or Virtual Environment.
  2. Install your required modules into your python environment (pip or other installation tool)
  3. Note the Python executable locations -Splunk Python location: (e.g. D:\Splunk\bin\python.exe) -External Python installation (e.g. D:\Splunk_Python_venv\p27_15\Scripts\python.exe)

Script Requirements:
1. Wrapper script: (see example_wrapper.py)
-Will work with Splunk python implementation. (No modules added outside of Splunk standard installation)
-Reference Intersplunk class
sys.path.append("/Splunk/Python-2.7/Lib/site-packages/splunk")
import Intersplunk
-Print output results from target script
2. Target script: (see example_target.py)
-Can use your modules or customizations
-Reference Intersplunk class
sys.path.append("/Splunk/Python-2.7/Lib/site-packages/splunk")
import Intersplunk
-Call Intersplunk.outputResults(results)
(results) is your result set and it must be in a format that can be consumed by Splunk.
Splunk Custom Search Command:
1. Create custom search command in commands.conf
[activatepython]
chuncked = false
filename = example_wrapper.py

Run Script from commandline:
\Splunk\etc\apps\myapp\bin>........\bin\splunk cmd python example_wrapper.py scripttarget=example_target.py

Run Script from SPLalt text

Example Files:

######################################################################################
############### following is the example_wrapper.py file #############################
######################################################################################
import os
import subprocess
import re
import sys
sys.path.append("/Splunk/Python-2.7/Lib/site-packages/splunk")
import Intersplunk


custompythonpath = 'D:\Splunk_Python_venv\p27_15\Scripts\python.exe'
splunkpythonpath = 'D:\Splunk\bin\python.exe'

os.environ['PYTHONPATH'] = custompythonpath
# from splunk web interface this is needed to get the arguments
# example SPL "| scriptwrapper scripttarget=example_target.py"
keywords, argvals = Intersplunk.getKeywordsAndOptions()
try:
    #pull in the arguments
    scripttarget = argvals['scripttarget']
    #prevent code injection: verify we aren't getting a file outside of our run directory 
    fileVerify=re.split('/|\*|\\\\|\'|\"', scripttarget)
    if os.path.isfile(fileVerify[0]):
        #create full path to your script
        my_process = os.path.join(os.getcwd(), scripttarget)
        #create subprocess be sure to pass stdin, stdout, stderr
        p = subprocess.Popen([custompythonpath, my_process, splunkpythonpath], stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
        # get the output
        output = p.communicate()[0]
        # push it out
        print output
    else:
        print ('Unable to verify requested action'.format(fileVerify))
except Exception as e:
    print(e)


######################################################################################
############### following is the example_target.py file #############################
######################################################################################


# pypyodbc has a dependency on environment variable 'windir' be sure 
# if using the the activate process that the class passes in the evirnoment variable on activate.
import sys
import pypyodbc
import sys
import os
sys.path.append("/Splunk/Python-2.7/Lib/site-packages/splunk")
import Intersplunk

def GetSqlResults(connectionString, sqlText):
    results = []
    try:
        db = pypyodbc.connect(connectionString)
        cursor = db.cursor()
        cursor.execute(sqlText)
        columns = [column[0] for column in cursor.description]
        #had to do the following to get splunk to accept the result set (I thought this would be simpler)
        for row in cursor.fetchall():
            result = {}
            for column in columns:
                result[column] = row[column]
            results.append(result)
        db.close()
    except Exception as e:
        print('Bad things happened when executing SQL: {0}'.format(e))

    return results


if __name__ == "__main__":
    connectionString = "Trusted_Connection=yes; Driver={SQL Server};server=greatstuffserver;database=greatstuffdatabase'
    sqlText = 'select id, name from greatstufftable'
    results = []
    results = GetSqlResults(connectionString, sqlText)
    #send results to Intersplunk (the wrapper class will 'print' the output)
    Intersplunk.outputResults(results)
    sys.exit(0)
Labels (2)

spunk_enthusias
Path Finder

Thank you! This may come in handy if I don't manage to find another option to keep the search command's process alive and end up having to manage one myself.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...