Splunk Dev

Python scripts importing modules from other apps

nvonkorff
Path Finder

In the process of creating a new app, I found I was getting cross-app module imports if another app has Python modules with the same name.

For example:

app1/bin/test_script.py
app1/bin/test_module.py

app2/bin/test_script.py
app2/bin/test_module.py

If test_script.py does an "import test_module", and I execute "app2/bin/test_script.py", it actually imports/executes "app1/bin/test_module.py"

I am writing a new app to import mail headers from audited Gmail accounts, and I used some of the same modules for the credential auth process from the Gsuite for Splunk app.

I found when I tried to auth credentials in the Gsuite for Splunk app, it imported and executed the "ga_authorize.py" from my app directory, not the GSuiteForSplunk/bin directory.

I added some logging into my copy of the "ga_authorize.py" script to print the sys.path, PYTHONPATH and the path of the script being executed to prove this:

2019-05-27 13:36:07,277 log_level=INFO pid=57869 tid=MainThread file="ga_authorize.py" function="handle_GET" line_number="51" version="GSuiteForSplunk.v1.2.3.b220" operation=build_url
2019-05-27 13:36:07,277 log_level=INFO pid=57869 tid=MainThread file="ga_authorize.py" function="<module>" line_number="32" version="GSuiteForSplunk.v1.2.3.b220" sys.path=['/opt/splunk/lib/python2.7/site-packages/splunk/appserver/mrsparkle', '/opt/splunk/bin', '/opt/splunk/lib/python2.7/site-packages', '/opt/splunk/lib/python27.zip', '/opt/splunk/lib/python2.7', '/opt/splunk/lib/python2.7/plat-linux2', '/opt/splunk/lib/python2.7/lib-tk', '/opt/splunk/lib/python2.7/lib-old', '/opt/splunk/lib/python2.7/lib-dynload', '/opt/splunk/bin', '/opt/splunk/etc/apps/FitbitAddonforSplunk/bin', '/opt/splunk/etc/apps/GSuiteForSplunk/bin', '/opt/splunk/etc/apps/Splunk_SA_CIM/bin', '/opt/splunk/etc/apps/Splunk_Security_Essentials/bin', '/opt/splunk/etc/apps/Splunk_TA_aws/bin', '/opt/splunk/etc/apps/Splunk_TA_nix/bin', '/opt/splunk/etc/apps/TA-GMail-audit/bin', '/opt/splunk/etc/apps/TA-GMail-audit_PROD/bin', '/opt/splunk/etc/apps/TA-cyberchef/bin', '/opt/splunk/etc/apps/TA-fitbit/bin', '/opt/splunk/etc/apps/TA-gmail-audit/bin', '/opt/splunk/etc/apps/alert_logevent/bin', '/opt/splunk/etc/apps/alert_webhook/bin', '/opt/splunk/etc/apps/base64/bin', '/opt/splunk/etc/apps/cis-controls-app-for-splunk/bin', '/opt/splunk/etc/apps/introspection_generator_addon/bin', '/opt/splunk/etc/apps/kapsch_roadside/bin', '/opt/splunk/etc/apps/lookup_editor/bin', '/opt/splunk/etc/apps/rest-storage-passwords-manager/bin', '/opt/splunk/etc/apps/search/bin', '/opt/splunk/etc/apps/splunk_app_addon-builder/bin', '/opt/splunk/etc/apps/splunk_archiver/bin', '/opt/splunk/etc/apps/splunk_instrumentation/bin', '/opt/splunk/etc/apps/splunk_monitoring_console/bin', '/opt/splunk/etc/system/bin', '/opt/splunk/etc/apps/TA-gmail-audit/bin/lib']

2019-05-27 13:36:07,277 log_level=INFO pid=57869 tid=MainThread file="ga_authorize.py" function="<module>" line_number="31" version="GSuiteForSplunk.v1.2.3.b220" PYTHONPATH=/opt/splunk/lib/python2.7/site-packages

2019-05-27 13:36:07,274 INFO SCRIPTPATH=/opt/splunk/etc/apps/TA-gmail-audit/bin/ga_authorize.py

You can see that the 'app/bin' directory of all apps is included in the sys.path and the name of the script being executed is 'TA-gmail-audit/bin/ga_authorize.py' not 'GSuiteforSplunk/bin/ga_authorize.py'

Is this expected behaviour? Is there any way to force Splunk/Python to only import/execute modules under the current app directory, and not pick up the same named modules in other app directories?

Keen to hear if anyone else has run into this issue, and if so, what was done to fix/workaround the issue.

Thanks.

Labels (2)
0 Karma
1 Solution

nvonkorff
Path Finder

It's not ideal, bit I decided to simply rename the Python script in my app that was conflicting with the other Python script of the same name in the other app.

i.e. instead of having ga_authorize.py in both apps, I have gmail_authorize.py in my app, so that it doesn't conflict with ga_authorize.py in the GSuite for Splunk app.

It doesn't fix or explain the underlying problem, but works around it enough for my needs.

View solution in original post

0 Karma

nvonkorff
Path Finder

To get around this issue, you can also specify the path of the module you wish to load, and load it using the 'imp' module:

import imp
modfile, pathname, description = imp.find_module('httplib2', ['/opt/splunk/etc/apps/TA-gmail-audit/bin/'])
httplib2 = imp.load_module('httplib2', modfile, pathname, description)

This is explained in the troubleshooting section of the Gmail Audit add-on

0 Karma

nvonkorff
Path Finder

It's not ideal, bit I decided to simply rename the Python script in my app that was conflicting with the other Python script of the same name in the other app.

i.e. instead of having ga_authorize.py in both apps, I have gmail_authorize.py in my app, so that it doesn't conflict with ga_authorize.py in the GSuite for Splunk app.

It doesn't fix or explain the underlying problem, but works around it enough for my needs.

0 Karma
Get Updates on the Splunk Community!

A Prelude to .conf25: Your Guide to Splunk University

Heading to Boston this September for .conf25? Get a jumpstart by arriving a few days early for Splunk ...

4 Ways the Splunk Community Helps You Prepare for .conf25

.conf25 is right around the corner, and whether you’re a first-time attendee or a seasoned Splunker, the ...

Enhance Your Splunk App Development: New Tools & Support

UCC FrameworkAdd-on Builder has been around for quite some time. It helps build Splunk apps faster, but it ...