Python for Scientific Computing and the SDK

[gwobben]

Hi,

I'm trying to build my own search command that makes use of pandas and numpy. That's why I'm trying to use the app "Python for Scientific Computing". When I run my code it works fine but when I try to wrap it in the Splunk Python SDK (to turn the script into a nice search command) I get errors.

The error is: ImportError: No module named ssl. I looked into the code of the "Python for Scientific Computing" and it seems that it's build without ssl. Is there any way to fix this? Without the ssl library the SDK and "Python for Scientific Computing" don't play nice..

Thanks!

[lshatzer]

This is because the version of Python running inside Python for Scientific Computing does not have SSL support.

I had to modify binding.py with a try catch block around the import ssl, set a property that it has SSL support, and places where it calls SSL, wrap it in an condition.

30c30,34
< import ssl
---
> try:
>     import ssl
>     hasssl = True
> except ImportError:
>     hasssl = False
548c552,555
<             sock = ssl.wrap_socket(sock)
---
>             if hasssl:
>                 sock = ssl.wrap_socket(sock)
>             else:
>                 raise ImportError('No SSL library found')
1331,1337c1338,1347
<             if key_file is not None: kwargs['key_file'] = key_file
<             if cert_file is not None: kwargs['cert_file'] = cert_file
< 
<             # If running Python 2.7.9+, disable SSL certificate validation
<             if sys.version_info >= (2,7,9) and key_file is None and cert_file is None:
<                 kwargs['context'] = ssl._create_unverified_context()
<             return httplib.HTTPSConnection(host, port, **kwargs)
---
>             if hasssl:
>                 if key_file is not None: kwargs['key_file'] = key_file
>                 if cert_file is not None: kwargs['cert_file'] = cert_file
> 
>                 # If running Python 2.7.9+, disable SSL certificate validation
>                 if sys.version_info >= (2,7,9) and key_file is None and cert_file is None:
>                     kwargs['context'] = ssl._create_unverified_context()
>                 return httplib.HTTPSConnection(host, port, **kwargs)
>             else:
>                 raise ImportError('No SSL library found')

[sjodle]

I've just submitted this as a pull request to the library. https://github.com/splunk/splunk-sdk-python/pull/205

[bmacias84]

@gwobben,

I am going to say yes, because I've done things such as incorporated NLTK (natural language library) into Splunk. The NTLK library required Numpy which requires pandas, sympy, scipy, etc. That said doing this is not easy since these are have arch-type dependancies and Splunk's python does not have all the dependancies such as Cython. If you want to incorporate these the best way if found is to build eggs using system python (same version of python running in your Splunk instance). Then inject the eggs into the search path for modules.

Example code:

import sys
import os
from platform import system

SPLUNK_HOME = os.environ.get('SPLUNK_HOME')
platform = system()

if platform == 'Darwin':
    platform = 'macosx'
egg_dir = SPLUNK_HOME + '/etc/apps/<yourAPP>/bin/'
for filename in os.listdir(egg_dir):
    filesegments = filename.split('-')
    if filename.endswith('.egg'):
        if len(filesegments) <= 3:
            sys.path.append(egg_dir + filename)
        else:
            if platform in filename:
                sys.path.append(egg_dir + filename)
import six
import numpy
import scipy
import nltk

So it is possible, but not fun. Last time I did something like this building eggs it took me like 8hrs working through dependancies and versions issues. Probably not the answer you are looking for.

Additional post I've written on this topic:
https://answers.splunk.com/answers/220196/import-non-native-python-libraries-into-splunk.html