topic Help with python wrapper! in Splunk Dev

Help with python wrapper!

jedatt01 — Mon, 28 Sep 2020 15:51:21 GMT

I'm trying to execute a python script as a scripted input. My script needs external modules to run so i've taken a wrapper script I found on the answers site and used it to call my external python and script. However, splunk does not pickup the output because the wrapper script immediately goes back to the command prompt before the other script finishes. See wrapper code below and example output. Please help!

OUTPUT
D:\Splunk\bin>splunk cmd python d:\Splunk\bin\scripts\globalapps\google_analytic
s_wrapper.py

D:\Splunk\bin>02/10/14 17:11:37 activeVisitors=889

Wrapper code

import os, sys
import time

DELETING ENV VARS

Remove problematic environmental variables if they exist.

for envvar in ("PYTHONPATH", "LD_LIBRARY_PATH"):
if envvar in os.environ:
del os.environ[envvar]

USING OS INTERPRETER

python_executable = "C:\Python27\python.exe"

real_script = os.path.join(os.path.dirname(file), "real_time_analytics.py")

os.execv(python_executable, [ python_executable, real_script ] )

Re: Help with python wrapper!

triest — Tue, 11 Feb 2014 04:09:39 GMT

Short Answer: Can you use subprocess.call([python_executable, real_script]) instead?

I confirmed that the script appears to do the right thing under OSX, so I dug out my Windows laptop and confirmed that it appears to fork and thus the wrapper script exits so it wouldn't see real_script's output

In researching msg 109179 on issue9148 (sorry it says I don't have enough karma to link to an external site): "on Windows, exec() does not really replace the current process. It creates a new process (with a new pid), and exits the current one" Thus this appears to be due to the way python behaves on windows.

I was able to get around the difference by using subprocess.call (I tested having python 2.7 call python 3.3 and printing the versions in both to confirm my understanding of the desired behavior.

Edit: As I processed things overnight, I realized I didn't point out that subprocess.call is different; it will fork, run the program, and then return to the original script. If you put a print statement after the subprocess.call it would execute the print statement (not tested, but based on my understanding). I think for your use case, this shouldn't matter since the original script is a wrapper script written explicitly for this purpose.

Re: Help with python wrapper!

jedatt01 — Tue, 11 Feb 2014 14:29:18 GMT

triest, I think i'm almost there. I'm now using subprocess.call and the script output shows correctly when I run in command prompt, but Splunk is still not sucking up the output from the subscript. In your edit you mentioned putting a print statement at the bottom but I couldn't get that to work. If I try to assign the subprocess.call to a variable, it just outputs an integer. Any ideas how to send the output back to the wrapper script for stdoutput?

Re: Help with python wrapper!

triest — Mon, 28 Sep 2020 15:51:58 GMT

The only thought I have off the top of my head would be to try:
subprocess.call([python_executable, real_script], stdout=sys.stdout)

I'm at work so I only have my Mac with me and on it that doesn't appear to change anything. I've not played with scripted inputs much yet so there may be some subtle aspect I'm not thinking of. I'll check back tonight (probably 9:00pm EST) and if there isn't anything I'll play with it for a couple of hours tonight and see if I can come up with something.

If you find a solution or workaround, please let me know!

Re: Help with python wrapper!

jedatt01 — Tue, 11 Feb 2014 18:27:56 GMT

well, turns out your first solution did work! I had some problems with my splunk instance and had to reinstall. Just to be thorough, i'll add that you need to import the subprocess module at the top of the wrapper script for this to work. Thanks for the help!