All Apps and Add-ons

Why is there no support for multi-threading with ML algorithms (n_jobs)?

frankwayne
Path Finder

Many (all?) of the sklearn algorithms support multi-threading using the n_jobs option. This is not exposed in Splunk, nor does fit or apply seem to use more than one thread. Why? Are there plans to do this?

For example, I was trying to improve the performance of RandomForestClassifier by making it multi-threaded, but n_jobs is not supported in RandomForestClassifier.py (not one of the 'ints' in out_params), nor is multi-threading an option in the UI's Settings tab for the MLTK.

0 Karma
1 Solution

astein_splunk
Splunk Employee
Splunk Employee

Hi! We do not expose these settings as the MLTK exposes machine learning as a first class citizen in Splunk's SPL paradigm and tries to stay true to SPL common behaviors. Splunk's SPL commands do not expose multithreaded options - all of that is abstracted by the SPL system. As the ML SPL commands (fit and apply) in the MLTK for the most part use only the search head resources, we want to be cognizant of the other potential production workloads on the shared Splunk infrastructure. If you are looking for massive scale machine learning I suggest looking at the Splunk MLTK Connector for Apache Spark (via Splunk beta) or the Splunk MLTK Container for TensorFlow (via PS) - both of which are leveraging non Splunk infrastructure for those large machine learning workflows.

View solution in original post

astein_splunk
Splunk Employee
Splunk Employee

Hi! We do not expose these settings as the MLTK exposes machine learning as a first class citizen in Splunk's SPL paradigm and tries to stay true to SPL common behaviors. Splunk's SPL commands do not expose multithreaded options - all of that is abstracted by the SPL system. As the ML SPL commands (fit and apply) in the MLTK for the most part use only the search head resources, we want to be cognizant of the other potential production workloads on the shared Splunk infrastructure. If you are looking for massive scale machine learning I suggest looking at the Splunk MLTK Connector for Apache Spark (via Splunk beta) or the Splunk MLTK Container for TensorFlow (via PS) - both of which are leveraging non Splunk infrastructure for those large machine learning workflows.

Get Updates on the Splunk Community!

Dashboard Studio Challenge - Learn New Tricks, Showcase Your Skills, and Win Prizes!

Reimagine what you can do with your dashboards. Dashboard Studio is Splunk’s newest dashboard builder to ...

Introducing Edge Processor: Next Gen Data Transformation

We get it - not only can it take a lot of time, money and resources to get data into Splunk, but it also takes ...

Take the 2021 Splunk Career Survey for $50 in Amazon Cash

Help us learn about how Splunk has impacted your career by taking the 2021 Splunk Career Survey. Last year’s ...