Python scripted inputs run with the wrong version of Python in Splunk 4.1.2

[joelshprentz]

We are running a few Python scripts as Splunk scripted inputs. This is an example stanza from our inputs.conf:

[script:///foo/bar/scripts/applicationFatalTraps.py]
interval = 60
index = main
sourcetype = log4j
source = script:applicationFatalTraps
disabled = 0

The first line of the script specifies which Python to run:

#!/usr/bin/python

When run from the Linux command line, the script logs this expected version of Python, which is installed as /usr/bin/python:

Python 2.3.4 (#1, Feb 18 2008, 17:16:53)
[GCC 3.4.6 20060404 (Red Hat 3.4.6-9)] on linux2

However, when run as scripted input from Splunk 4.1.2, the script logs this newer version of Python :

Python version = "2.6.4 (r264:75706, Apr  8 2010, 00:28:45) 
[GCC 3.4.6 20060404 (Red Hat 3.4.6-3)]"

Needless to say, the Python version difference breaks something in the script.

Question: Why is Splunk running the script with the wrong version of Python?

[Jason]

Hint: to get at Splunk's version of python, run the following:

$SPLUNK_HOME/bin/splunk cmd python

So, to test a script in it,

$SPLUNK_HOME/bin/splunk cmd python yourscript.py

[Lowell · Accepted solution]

Splunk't bundled version of python is always used for python scripts.

This is like the different between running ./applicationFatalTraps.py vs running python applicationFatalTraps.py. With the first, the kernel decides because of that first !# line, where as with the second, it depends on which python first occurs in your path. And in the case of splunk, their own python is always first. (This may not be 100% technically accurate, but I think it gives you the right idea.)

---

To use a different python interpreter, simply create small shell script:

Example /foo/bar/scripts/applicationFatalTraps.sh:

 #!/bin/sh
 unset PYTHONPATH
 unset LD_LIBRARY_PATH
 exec /usr/bin/python /foo/bar/scripts/applicationFatalTraps.py

Or, if you need a more platform independent solution, you can use a small python script to launch a different interpreter.

import os, sys
# Remove problematic environmental variables if they exist.
for envvar in ("PYTHONPATH", "LD_LIBRARY_PATH"):
    if envvar in os.environ:
        del os.environ[envvar]

python_executable = "/usr/bin/python"
real_script = "/opt/splunk/etc/apps/myapp/bin/applicationFatalTraps.py"

os.execv(python_executable, [ python_executable, real_script ] + sys.argv[1:])

This was copied from jrodman's post. Since it was so short I figured I'd just copy it here.

---

Update: Just another thought. What happens if you simply rename ApplicationFatalTraps.py to ApplicationFatalTraps (no extension). It's possible that splunk will, in that case issue an OS level system execution call and therefore let the OS decide which interpreter to run. You might still run into trouble because of some environmental variables, such as $PYTHONPATH... That thought just occurred to me. But I haven't tried, so who knows.

[jrodman]

FWIW, in both recipes you probably want to consider manipulating the PATH variable as well, to put the things you'll want to use higher on the path than splunk's bin (or dropping the splunk bin dir entirely if that matches your goals), but that's harder to encapsulate in an example.

[Lowell]

Did you upgrade your local python install, or are you just using splunk's bundled python distro.... it wasn't clear from your comment... BTW, I don't have access to update the docs. However, you could simply email docs@splunk.com with your request and add a link to this page. The docs team has been very responsive to all my nagging about various little doc issues here and there. 😉

[joelshprentz]

Thanks, Lowell.

We reviewed your excellent response and decided to upgrade our Python script to run with Python 2.6, the version currently used by Splunk.

May I recommend that you update the documentation to reveal Splunk's special handling of Python scripts?