Splunk Machine Learning Toolkit install additional Python packages

[wfskmoney]

I am researching to implement a LightGBM model using Skicit Learn in Python. Is it possible to install LightGBM Python packages so it can be used through the ML-SPL API ?

[badarsebard]

I've recently released a new app called PyDen on Splunkbase which allows developers to create Python virtual environments of varying versions (2.7, 3.5 - 3.7) that can be used to run custom scripts instead of the built-in Python Splunk uses. There is also a pip command available from the search bar that lets you install PyPI packages.

[datasearchninja]

The Splunk ML toollkit is sitting on top of a copy of Anaconda Python 2.7 sitting in Splunk_SA_Scientific_Python_linux_x86_64 (or other relevant OS/arch directory)

It is possible to just download a copy of Anaconda 2.7 and then copy over the relevant modules. From my notes it looks like https://repo.anaconda.com/archive/Anaconda2-4.0.0-Linux-x86_64.sh out of https://repo.anaconda.com/archive/ is a good place to start. Install it on its own, and then install the relevant module, then copy the directory, and deps over to the splunk app module directory Splunk_SA_Scientific_Python_linux_x86_64/bin/linux_x86_64/lib/python2.7. When testing you need to make sure you execute via the splunk python to call the anaconda python as the install uses libraries from both.

If you want to install them in a different Splunk app than the ML app, you will need to copy over and perhaps modify the exec_anaconda.py as per the instructions in the SA README. I have successfully done this to add pandas and pyarrow to a custom application which uses the base numpy package out of Splunk_SA_Scientific_Python_linux_x86_64.

To use Splunk_SA_Scientific_Python, copy exec_anaconda.py into your app's bin directory. At the top of your custom search command, add the following preamble:

#!/usr/bin/python    import
exec_anaconda   
exec_anaconda.exec_anaconda()    # Put
the rest of your imports below, e.g.: 
import numpy as np

[wfskmoney]

Thank you - is it also possible to use the newest anaconda environment with python 3.6 to invoke python classes?

[kptae]

I also want to know how to add python package to SMLT.