Importing Python module to Splunk

[ericyeh1995]

Using Linux I had installed freeopcua module for both my python 2.7 and python 3.5 (using pip and pip3). For what I found online people don't install new Python module within Splunk, therefore there is no way to know if I can install a module or is there a better way to import module. Needless to say, the script does not run and no data is imported to Splunk.

There are 3 Python in play here: the python 2.7 and python 3.5 that comes with Ubuntu, and Splunk's python 2.7. The Python in Splunk can not be updated to version 3. In short, the script works on both my Python 2.7 and Python 3.5, but I couldn't get Splunk's Python to run my script due to module error. My modifications include adding a module to sys.path and subprocess but doesn't seem to work. Below is a more detailed report:

My default debugging setup:
In terminal I typed in:

splunk cmd ./job.sh

The job.sh file called my .py script:

$SPLUNK_HOME/bin/python $SPLUNK_HOME/etc/apps/opc_ua_test/bin/splunk_opcua_client.py

Or it might call a subprocess in some tests

The script is a OPC UA client that gets information from a OPC UA server.

The terminal output error: Module not imported
(see error message 1 below)

Than I make some modifications, one variable at a time:
=> Modify job.sh to run the script using Ubuntu's python and called:

splunk cmd ./job.sh

job.sh body for reference:

/usr/bin/python $SPLUNK_HOME/etc/apps/opc_ua_test/bin/splunk_opcua_client.py

Terminal output undefined symbol error, see error message 2 below.

=> Modify job.sh to run the script using python3:
Error output error message 3 (again, shown below)

=> Going back to default and add the module path to sys.path: error message 1

=> Have job.sh call a python subprocess. The subprocess call the script with python 2: error message 2

=> The subprocess call the script with python 3: error message 3

Even though there is an error, the message I printed out in the subprocess script is fed into Splunk.

=> The weird part is it works when I call the subprocess script with Splunk's python:
on terminal:

/opt/splunk/bin/python script_subprocess.py

It becomes clear to me that Splunk is running its own shell, and I have no clue on how to break out of it. Any suggestions on how to fix this will be nice!

---

Error messages:
1.

Traceback (most recent call last):
  File "/opt/splunk/etc/apps/opc_ua_test/bin/splunk_opcua_client.py", line 4, in <module>
    from opcua import Client
ImportError: No module named opcua

2.

Traceback (most recent call last):
  File "/opt/splunk/etc/apps/opc_ua_test/bin/splunk_opcua_client.py", line 4, in <module>
    from opcua import Client
  File "/usr/local/lib/python2.7/dist-packages/opcua/__init__.py", line 9, in <module>
    from opcua.client.client import Client
  File "/usr/local/lib/python2.7/dist-packages/opcua/client/client.py", line 19, in <module>
    from opcua.common.structures_generator import StructGenerator
  File "/usr/local/lib/python2.7/dist-packages/opcua/common/structures_generator.py", line 10, in <module>
    from lxml import objectify
ImportError: /opt/splunk/lib/python2.7/site-packages/lxml/objectify.so: undefined symbol: PyUnicodeUCS2_Compare

3.

Error in sitecustomize; set PYTHONVERBOSE for traceback:
AttributeError: module 'sys' has no attribute 'setdefaultencoding'
Traceback (most recent call last):
  File "/opt/splunk/etc/apps/opc_ua_test/bin/splunk_opcua_client.py", line 4, in <module>
    from opcua import Client
  File "/usr/local/lib/python3.5/dist-packages/opcua/__init__.py", line 9, in <module>
    from opcua.client.client import Client
  File "/usr/local/lib/python3.5/dist-packages/opcua/client/client.py", line 17, in <module>
    from opcua.crypto import security_policies
  File "/usr/local/lib/python3.5/dist-packages/opcua/crypto/security_policies.py", line 6, in <module>
    from opcua.crypto import uacrypto
  File "/usr/local/lib/python3.5/dist-packages/opcua/crypto/uacrypto.py", line 3, in <module>
    from cryptography import x509
  File "/usr/lib/python3/dist-packages/cryptography/x509/__init__.py", line 7, in <module>
    from cryptography.x509.base import (
  File "/usr/lib/python3/dist-packages/cryptography/x509/base.py", line 14, in <module>
    from cryptography.hazmat.primitives.asymmetric import dsa, ec, rsa
  File "/usr/lib/python3/dist-packages/cryptography/hazmat/primitives/asymmetric/rsa.py", line 14, in <module>
    from cryptography.hazmat.backends.interfaces import RSABackend
  File "/usr/lib/python3/dist-packages/cryptography/hazmat/backends/__init__.py", line 7, in <module>
    import pkg_resources
  File "/opt/splunk/lib/python2.7/site-packages/pkg_resources.py", line 33
    def _bypass_ensure_directory(name, mode=0777):
                                               ^
SyntaxError: invalid token

[dgawande]

For the error mentioned in 3rd point, try the solution posted in the following post.

https://answers.splunk.com/answers/701785/why-is-my-bash-scripted-input-failing-on-ubuntu-wh.html

[ericyeh1995]

I'd find my way out by manually copying the module folder into the folder with my script.