Why is there no support for multi-threading with ML algorithms (n_jobs)?

[frankwayne]

Many (all?) of the sklearn algorithms support multi-threading using the n_jobs option. This is not exposed in Splunk, nor does fit or apply seem to use more than one thread. Why? Are there plans to do this?

For example, I was trying to improve the performance of RandomForestClassifier by making it multi-threaded, but n_jobs is not supported in RandomForestClassifier.py (not one of the 'ints' in out_params), nor is multi-threading an option in the UI's Settings tab for the MLTK.

[astein_splunk · Accepted solution]

Hi! We do not expose these settings as the MLTK exposes machine learning as a first class citizen in Splunk's SPL paradigm and tries to stay true to SPL common behaviors. Splunk's SPL commands do not expose multithreaded options - all of that is abstracted by the SPL system. As the ML SPL commands (fit and apply) in the MLTK for the most part use only the search head resources, we want to be cognizant of the other potential production workloads on the shared Splunk infrastructure. If you are looking for massive scale machine learning I suggest looking at the Splunk MLTK Connector for Apache Spark (via Splunk beta) or the Splunk MLTK Container for TensorFlow (via PS) - both of which are leveraging non Splunk infrastructure for those large machine learning workflows.