In Splunk Enterprise, can you help me call an external Python3 via Script?

blangrill — Tue, 02 Oct 2018 18:38:28 GMT

Splunk still uses Python 2.7 internally but has the ability to call external scripts to generate data via Scripted Inputs

I would like to pull data using an external API which has a Python 3 library. I have installed Python 3 to a separate place on the file system and written a Windows batch script, which calls this and invokes the Python 3 API. When run from the OS, this generates the data I would like, but if I try to add this batch script as an input to Splunk, I get an error along the lines of the following:

ERROR ExecProcessor - message from "e:\splunk\bin\scripts\test.bat" Fatal Python error: Py_Initialize: unable to load the file system codec 
ERROR ExecProcessor - message from "e:\splunk\bin\scripts\test.bat" File "E:\splunk\Python-2.7\Lib\encodings\__init__.py", line 123 
ERROR ExecProcessor - message from "e:\splunk\bin\scripts\test.bat" raise CodecRegistryError,\ 
ERROR ExecProcessor - message from "e:\splunk\bin\scripts\test.bat" ^ 
ERROR ExecProcessor - message from "e:\splunk\bin\scripts\test.bat" SyntaxError: invalid syntax

I get a similar error even if I minimally set the batch script to only contain a minimal statement.

<path to python3>\python.exe -c 'print("Hello")'

This implies the problem is something from Python 2 being passed to the Python 3 environment since the Splunk script call is ultimately from that.

I asked Splunk support about this and was told it "isn't supported" and was directed to contact professional services which seems like overkill for what is likely just an environment issue.

Is there a way to wrap the python commands such that this still works?

As of now my workaround is to have the batch file called as a scheduled task in Windows and write the results to a file which is then monitored by Splunk. Additionally, at some point Splunk will make the jump internally to Python 3 and likely people will have the reverse problem with older Python 2.X libraries.

Re: In Splunk Enterprise, can you help me call an external Python3 via Script?

blangrill — Wed, 10 Oct 2018 14:41:49 GMT

The answer is quite simple. Adding -E to the Python3 call tells it to ignore other PYTHONPATH variables and it is able to run successfully.

The batch file I am calling as a Splunk scripted input now looks like:

@ECHO off
C:\Python37\python.exe -E "<path to python3 script>"

Re: In Splunk Enterprise, can you help me call an external Python3 via Script?

badarsebard — Sun, 30 Dec 2018 04:59:47 GMT

There's a new app called PyDen which would allow for an alternative approach to this problem. The app allows a developer to create Python virtual environments of any version (2.7, 3.5+) for use by scripts run by Splunk. This would allow you to do simply use the Python 3 script you've written as the input script instead of using a wrapper.

topic Re: In Splunk Enterprise, can you help me call an external Python3 via Script? in Installation

In Splunk Enterprise, can you help me call an external Python3 via Script?

Re: In Splunk Enterprise, can you help me call an external Python3 via Script?

Re: In Splunk Enterprise, can you help me call an external Python3 via Script?