topic Re: Scripted Input - Python Module Error in Getting Data In

Scripted Input - Python Module Error

nakiamatthews — Thu, 07 Feb 2019 02:27:27 GMT

I have a python script that pulls data from an SFTP source and writes the data to a file (myScript.py). The script imports two modules that I installed. As such, I wrote a bash wrapper script to call my script and run it with standard python (myScriptLauncher.sh).

I can run the bash script from the command line fine. However, when I try to run it using Splunk (./splunk cmd /opt/splunk/etc/apps/myApp/bin/myScriptLauncher.sh), I get a module import error.

I'm not sure what I am doing wrong. Anyone have any suggestions?

error

Traceback (most recent call last):
  File "./myScript.py", line 4, in <module>
    import pysftp
ImportError: No module named pysftp

myScript.py

#!/usr/bin/env python

import json
import pysftp
import openpyxl

REST OF CODE GOES HERE

myScriptLauncher.sh

#!/bin/bash

echo myScript is starting >&2

cd $( dirname "${BASH_SOURCE[0]}" )

exec /usr/bin/python ./myScript.py

Re: Scripted Input - Python Module Error

kamlesh_vaghela — Thu, 07 Feb 2019 05:48:21 GMT

@nakiamatthews

I think you have installedpysftp files in local python. So you should add path in your myScript.py script OR you have to copy those file into splunk lib.

See below ref link for your more info:

https://answers.splunk.com/answers/222934/modular-input-that-need-additional-python-modules.html

https://answers.splunk.com/answers/220196/import-non-native-python-libraries-into-splunk.html#answer-222585
https://answers.splunk.com/answers/8/can-i-add-python-modules-to-the-splunk-environment.html

Re: Scripted Input - Python Module Error

nakiamatthews — Thu, 07 Feb 2019 17:45:27 GMT

I was under the impression that adding the modules to the splunk lib was not future proof - i.e. it will break whenever you upgrade Splunk.

Re: Scripted Input - Python Module Error

nakiamatthews — Thu, 07 Feb 2019 18:12:31 GMT

Also, I thought the point of the wrapper script was to call the .py script using standard python?

Re: Scripted Input - Python Module Error

badarsebard — Tue, 26 Feb 2019 05:46:35 GMT

Creating a wrapper to call a Python installation on the host has been the conventional Splunk wisdom for years, as was bundling needed modules into the app. Both these approaches have numerous problems though. Using the wrapper is typically temperamental to permissions, environment variables, and in the case of the default system Python possibly versions of Python even older than 2.7.

My suggestion is to check out a new app I've written called PyDen which allows you to create Python virtual environments and install PyPI packages to the environment. You can then use these environments to execute custom commands and scripted inputs.

This provides three key advantages:

You can pick the version of Python you'd like to write in, including 3.5 to 3.7
Create an isolated environment that won't interfere with imports in other apps (this can happen when packages are bundled into apps)
Install any package available on PyPI to the environment