Splunk Application cannot load script with Splunk Enterprise Security enabled

[max-ipinfo]

I maintain IPinfo's Splunk App: https://splunkbase.splunk.com/app/4070

Our customers have recently reported that our application doesn't work when Splunk Enterprise Security is enabled.

For context, our application uses one of two modes to interact with our data:
1) queries our API directly
2) downloads our datasets locally using a public HTTPS endpoint

The failure only happens in the second mode, when we have to make REST calls to coordinate the download of our data.

One key finding in my early investigation is that our Splunk application communicates using raw non-SSL-verified HTTPS requests (i.e. using the requests Python library with verify=False), authenticated by session keys.

Splunk Entreprise Security seems to prevent these types of communication. To bypass this restriction, I converted everything over to the Splunk Python SDK, which bypasses all of these SSL issues.

I quickly realized that, to leverage the Splunk Python SDK in all scenarios and with consistency, it would just be easier to use bearer tokens throughout, so the second change I made was leveraging bearer tokens for REST communications.

Despite these two changes, the application still doesn't work with Splunk Entreprise Security enabled. It works without a problem when it is disabled (for example, when testing in the Docker Splunk dev environment).

I've also tried to build a simple debug handler based on splunk.rest.BaseRestHandler. When I try to call it directly with Splunk Entreprise Security enabled, I get the following error:

ERROR - HTTP 500 Error starting: Can't load script "/opt/splunk/etc/apps/ipinfo_app/bin/debug_endpoint.py" -- Error starting: Can't load script "/opt/splunk/etc/apps/ipinfo_app/bin/debug_endpoint.py" 

I haven't been able to track this particular error in Splunk forums or other forums on the Internet. 

If anyone has insight on this problem, I would appreciate any help. Thank you!

[max-ipinfo]

Sorry for the delay @livehybrid.

We just managed to get access to a customer's Splunk environment this week and it was very productive!

Here are our findings:

• we validated that our Splunk app works with Splunk Enterprise Security, on a standalone node

• we were able to recreate the bug in their clustering setup:

  • we found out that not all of our Python scripts were executable, preventing execution in that context (Can't load script error)

  • the source of the download errors was finally root-caused: Splunk Enterprise Security hijacks the Python module path order. So when we were trying to import our application's bin/utils.py in our own code, it was trying to import /opt/splunk/etc/apps/SA-ThreatIntelligence/bin/utils.py
    
    When we overrode sys.path in our script in the customer environment, the application worked again. The simplest work-around is to prefix all of our script files with ipinfo_ to prevent module name collision.
    
    We still feel that the Python module path hijacking should not be happening. Not sure if this is a bug that Splunk Platform teams should fix. If I need to file a bug somewhere, let me know!

• we noticed that we don't have to assign a bearer token to the Splunk admin user for our REST API calls to work with the Python SDK. We can use another user (e.g. ipinfo_admin) with a restricted set of permissions. We are still trying to figure out what the smallest amount of permissions are required for things to work.

Next step is applying all of the fixes above and see if it resolves our customers' problems. I'll reach out in this thread if new issues pop up.

[livehybrid]

Hi @max-ipinfo 

Were you able to find anything in $SPLUNK_HOME/var/log/splunkd.log relating to this file and the 500 error?

You could also try running  

$SPLUNK_HOME/bin/splunk cmd python3 /opt/splunk/etc/apps/ipinfo_app/bin/debug_endpoint.py

 to check that the python file has no syntax errors - you might not get an output if it works, but you may well get an error if there is an issue..

Its also worth checking the ownership and permissions on this file on the filesystem.

If you still have no success feel free to share the python file contents and we can continue to debug with you.

Please let me know how you get on and consider accepting this answer or adding karma this answer if it has helped.
Regards

Will

[max-ipinfo]

Thanks @livehybrid . I'll ask our customers to try out your suggestions and will report back.

I really appreciate your help!