Setting up a python virtual environment for developing Splunk applications

[thorsten_osaka]

I am new to Splunk and want to write my own MLTK classes/functions. I want to test my code locally in Anaconda or PyCharm. Therefor I would like to set up a virtual python environment that is identical to the one used in Splunk. Something like the output of a "pip freeze". Have searched the internet, but could not find a list python packages with versions needed to set up this environment. Any ideas where I can find that?

[DavidHourani]

Hi @thorsten_osaka,

Seems like you're looking for the Splunk Python SDK. Have a look here for the environment and some tutorials :

http://dev.splunk.com/python

If it's specifically for MLTK please check here :
https://docs.splunk.com/Documentation/MLApp/4.2.0/API/Overview

    To add a custom algorithm to the Splunk Machine Learning Toolkit, you must write a python class and register it to the ML-SPL algorithm list. The algorithm class must implement certain methods which are outlined in the BaseAlgo class in $SPLUNK_HOME/etc/apps/Splunk_ML_Toolkit/bin/base.py.

    These are the high level steps to import a new algorithm to the Splunk Machine Learning Toolkit:

   1- Register the algorithm.
   2- Create a python script file (.py file) for the algorithm.
   3-  Write an algorithm class.

Cheers,
David

[thorsten_osaka]

Dear David,

thanks for the quick answer. This is exactly what I want to do. Implementing the BaseAlgo class.
My problem is: Where/How do I develop my code? My plan was to set up a virtual environment in Anaconda or PyCharm which is identical to the splunk environment (python 2.7, scikit 0.17, scipy ...) I just could not find a comprehensive list of dependencies. I also look into PyDen, but could not get it to work yet. Also, I wonder: What is best practice for developing and testing ML classes?

Cheers,
Thorsten

[badarsebard]

PyDen is built around executing Python code in Splunk that is either a) a different version than the one bundled with Splunk or b) utilizes modules not included in that bundle.

If your use case is to develop Python code for use with MLTK I'd suggest downloading the Python for Scientific Computing Add On. This add on has an Anaconda interpreter and several common Python libraries for ML, and when the MLTK executes its Python code it uses this interpreter instead of the one built into Splunk. Once downloaded you can set the interpreter inside the add on as the project interpreter for Pycharm.