topic Re: Splunk Machine Learning Toolkit install additional Python packages in All Apps and Add-ons

Splunk Machine Learning Toolkit install additional Python packages

wfskmoney — Mon, 17 Sep 2018 15:05:27 GMT

I am researching to implement a LightGBM model using Skicit Learn in Python. Is it possible to install LightGBM Python packages so it can be used through the ML-SPL API ?

Re: Splunk Machine Learning Toolkit install additional Python packages

kptae — Thu, 20 Sep 2018 01:44:10 GMT

I also want to know how to add python package to SMLT.

Re: Splunk Machine Learning Toolkit install additional Python packages

datasearchninja — Tue, 29 Sep 2020 21:20:21 GMT

The Splunk ML toollkit is sitting on top of a copy of Anaconda Python 2.7 sitting in Splunk_SA_Scientific_Python_linux_x86_64 (or other relevant OS/arch directory)

It is possible to just download a copy of Anaconda 2.7 and then copy over the relevant modules. From my notes it looks like https://repo.anaconda.com/archive/Anaconda2-4.0.0-Linux-x86_64.sh out of https://repo.anaconda.com/archive/ is a good place to start. Install it on its own, and then install the relevant module, then copy the directory, and deps over to the splunk app module directory Splunk_SA_Scientific_Python_linux_x86_64/bin/linux_x86_64/lib/python2.7. When testing you need to make sure you execute via the splunk python to call the anaconda python as the install uses libraries from both.

If you want to install them in a different Splunk app than the ML app, you will need to copy over and perhaps modify the exec_anaconda.py as per the instructions in the SA README. I have successfully done this to add pandas and pyarrow to a custom application which uses the base numpy package out of Splunk_SA_Scientific_Python_linux_x86_64.

To use Splunk_SA_Scientific_Python, copy exec_anaconda.py into your app's bin directory. At the top of your custom search command, add the following preamble:

#!/usr/bin/python    import
exec_anaconda   
exec_anaconda.exec_anaconda()    # Put
the rest of your imports below, e.g.: 
import numpy as np

Re: Splunk Machine Learning Toolkit install additional Python packages

wfskmoney — Mon, 22 Oct 2018 12:43:57 GMT

Thank you - is it also possible to use the newest anaconda environment with python 3.6 to invoke python classes?

Re: Splunk Machine Learning Toolkit install additional Python packages

badarsebard — Sun, 30 Dec 2018 05:20:17 GMT

I've recently released a new app called PyDen on Splunkbase which allows developers to create Python virtual environments of varying versions (2.7, 3.5 - 3.7) that can be used to run custom scripts instead of the built-in Python Splunk uses. There is also a pip command available from the search bar that lets you install PyPI packages.