Deployment Architecture

Setting up a python virtual environment for developing Splunk applications

thorsten_osaka
Engager

I am new to Splunk and want to write my own MLTK classes/functions. I want to test my code locally in Anaconda or PyCharm. Therefor I would like to set up a virtual python environment that is identical to the one used in Splunk. Something like the output of a "pip freeze". Have searched the internet, but could not find a list python packages with versions needed to set up this environment. Any ideas where I can find that?

Tags (1)
0 Karma

DavidHourani
Super Champion

Hi @thorsten_osaka,

Seems like you're looking for the Splunk Python SDK. Have a look here for the environment and some tutorials :

http://dev.splunk.com/python

If it's specifically for MLTK please check here :
https://docs.splunk.com/Documentation/MLApp/4.2.0/API/Overview

    To add a custom algorithm to the Splunk Machine Learning Toolkit, you must write a python class and register it to the ML-SPL algorithm list. The algorithm class must implement certain methods which are outlined in the BaseAlgo class in $SPLUNK_HOME/etc/apps/Splunk_ML_Toolkit/bin/base.py.

    These are the high level steps to import a new algorithm to the Splunk Machine Learning Toolkit:

   1- Register the algorithm.
   2- Create a python script file (.py file) for the algorithm.
   3-  Write an algorithm class.

Cheers,
David

thorsten_osaka
Engager

Dear David,

thanks for the quick answer. This is exactly what I want to do. Implementing the BaseAlgo class.
My problem is: Where/How do I develop my code? My plan was to set up a virtual environment in Anaconda or PyCharm which is identical to the splunk environment (python 2.7, scikit 0.17, scipy ...) I just could not find a comprehensive list of dependencies. I also look into PyDen, but could not get it to work yet. Also, I wonder: What is best practice for developing and testing ML classes?

Cheers,
Thorsten

0 Karma

badarsebard
Communicator

PyDen is built around executing Python code in Splunk that is either a) a different version than the one bundled with Splunk or b) utilizes modules not included in that bundle.

If your use case is to develop Python code for use with MLTK I'd suggest downloading the Python for Scientific Computing Add On. This add on has an Anaconda interpreter and several common Python libraries for ML, and when the MLTK executes its Python code it uses this interpreter instead of the one built into Splunk. Once downloaded you can set the interpreter inside the add on as the project interpreter for Pycharm.

0 Karma
Get Updates on the Splunk Community!

Continuing Innovation & New Integrations Unlock Full Stack Observability For Your ...

You’ve probably heard the latest about AppDynamics joining the Splunk Observability portfolio, deepening our ...

Monitoring Amazon Elastic Kubernetes Service (EKS)

As we’ve seen, integrating Kubernetes environments with Splunk Observability Cloud is a quick and easy way to ...

Cloud Platform & Enterprise: Classic Dashboard Export Feature Deprecation

As of Splunk Cloud Platform 9.3.2408 and Splunk Enterprise 9.4, classic dashboard export features are now ...