All Apps and Add-ons

Why is there no support for multi-threading with ML algorithms (n_jobs)?

frankwayne
Path Finder

Many (all?) of the sklearn algorithms support multi-threading using the n_jobs option. This is not exposed in Splunk, nor does fit or apply seem to use more than one thread. Why? Are there plans to do this?

For example, I was trying to improve the performance of RandomForestClassifier by making it multi-threaded, but n_jobs is not supported in RandomForestClassifier.py (not one of the 'ints' in out_params), nor is multi-threading an option in the UI's Settings tab for the MLTK.

0 Karma
1 Solution

astein_splunk
Splunk Employee
Splunk Employee

Hi! We do not expose these settings as the MLTK exposes machine learning as a first class citizen in Splunk's SPL paradigm and tries to stay true to SPL common behaviors. Splunk's SPL commands do not expose multithreaded options - all of that is abstracted by the SPL system. As the ML SPL commands (fit and apply) in the MLTK for the most part use only the search head resources, we want to be cognizant of the other potential production workloads on the shared Splunk infrastructure. If you are looking for massive scale machine learning I suggest looking at the Splunk MLTK Connector for Apache Spark (via Splunk beta) or the Splunk MLTK Container for TensorFlow (via PS) - both of which are leveraging non Splunk infrastructure for those large machine learning workflows.

View solution in original post

astein_splunk
Splunk Employee
Splunk Employee

Hi! We do not expose these settings as the MLTK exposes machine learning as a first class citizen in Splunk's SPL paradigm and tries to stay true to SPL common behaviors. Splunk's SPL commands do not expose multithreaded options - all of that is abstracted by the SPL system. As the ML SPL commands (fit and apply) in the MLTK for the most part use only the search head resources, we want to be cognizant of the other potential production workloads on the shared Splunk infrastructure. If you are looking for massive scale machine learning I suggest looking at the Splunk MLTK Connector for Apache Spark (via Splunk beta) or the Splunk MLTK Container for TensorFlow (via PS) - both of which are leveraging non Splunk infrastructure for those large machine learning workflows.

Get Updates on the Splunk Community!

Optimize Cloud Monitoring

  TECH TALKS Optimize Cloud Monitoring Tuesday, August 13, 2024  |  11:00AM–12:00PM PST   Register to ...

What's New in Splunk Cloud Platform 9.2.2403?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.2.2403! Analysts can ...

Stay Connected: Your Guide to July and August Tech Talks, Office Hours, and Webinars!

Dive into our sizzling summer lineup for July and August Community Office Hours and Tech Talks. Scroll down to ...