All Apps and Add-ons

Splunk Machine Learning Toolkit install additional Python packages

wfskmoney
Path Finder

I am researching to implement a LightGBM model using Skicit Learn in Python. Is it possible to install LightGBM Python packages so it can be used through the ML-SPL API ?

0 Karma

badarsebard
Communicator

I've recently released a new app called PyDen on Splunkbase which allows developers to create Python virtual environments of varying versions (2.7, 3.5 - 3.7) that can be used to run custom scripts instead of the built-in Python Splunk uses. There is also a pip command available from the search bar that lets you install PyPI packages.

0 Karma

datasearchninja
Communicator

The Splunk ML toollkit is sitting on top of a copy of Anaconda Python 2.7 sitting in Splunk_SA_Scientific_Python_linux_x86_64 (or other relevant OS/arch directory)

It is possible to just download a copy of Anaconda 2.7 and then copy over the relevant modules. From my notes it looks like https://repo.anaconda.com/archive/Anaconda2-4.0.0-Linux-x86_64.sh out of https://repo.anaconda.com/archive/ is a good place to start. Install it on its own, and then install the relevant module, then copy the directory, and deps over to the splunk app module directory Splunk_SA_Scientific_Python_linux_x86_64/bin/linux_x86_64/lib/python2.7. When testing you need to make sure you execute via the splunk python to call the anaconda python as the install uses libraries from both.

If you want to install them in a different Splunk app than the ML app, you will need to copy over and perhaps modify the exec_anaconda.py as per the instructions in the SA README. I have successfully done this to add pandas and pyarrow to a custom application which uses the base numpy package out of Splunk_SA_Scientific_Python_linux_x86_64.

To use Splunk_SA_Scientific_Python, copy exec_anaconda.py into your app's bin directory. At the top of your custom search command, add the following preamble:

#!/usr/bin/python    import
exec_anaconda   
exec_anaconda.exec_anaconda()    # Put
the rest of your imports below, e.g.: 
import numpy as np
0 Karma

wfskmoney
Path Finder

Thank you - is it also possible to use the newest anaconda environment with python 3.6 to invoke python classes?

0 Karma

kptae
New Member

I also want to know how to add python package to SMLT.

0 Karma
Get Updates on the Splunk Community!

Using Machine Learning for Hunting Security Threats

REGISTER NOW Seeing the exponential hike in global cyber threat spectrum, organizations are now striving more ...

Security Highlights | November 2022 Newsletter

 November 2022 2022 Gartner Magic Quadrant for SIEM: Splunk Named a Leader for the 9th Year in a RowSplunk is ...

Platform Highlights | November 2022 Newsletter

 November 2022 Skill Up on Splunk with our New Builder Tech Talk SeriesCan you build it? Yes you can! *play ...