How to export/import/share ML models between Splunk instances and external system?

[melonman]

Hi Experts!

I have an Splunk instances that keeps logs and created a machine learning model of a type of regression.
Now I want to send the model to heavy forwarder that is running on the edge collecting logs from end points/devices.

Batch Learning on indexer and Online Scoring on Heavy Forwarder.

Source -> Heavy Forwarder with MLTK --- network---> Indexers/SHs with MLTK

Question:
- Is there any way to automatically or manually to export/import the model to different Splunk instances?
- Is there a way to share the model between Splunk and 3rd party applications (like PMML or similar)

Thank you!

[aljohnson_splun]

Hi @melonman, there are maybe a few confusions I want to clear up, just in case.

Training a model in the MLTK will create a model, which is stored and moved around as a lookup. What this means, is that you only are going to be using this model in the context of a search itself (with the apply command). In your question, you mention a heavy forwarder, which, unless is also a search peer, will never have any reason to interact with the MLTK model.

If, your heavy forwarder does happen to also be a search peer, then it will be relevant. In that case, there are a few other things to keep in mind. In a normal apply job, the results of the search are collected at the search head like a reporting command, and the models is applied on the results in one location (the search head).

If you look in mlspl.conf, you'll see there is also an option to turn streaming_apply=true. If you choose to enable this capability, then the apply command will become a streaming command and the model will be distributed to the search peers via a search bundle configuration. In this case, the model is applied in parallel across the search peers.

Note however, in both scenarios, the presence of a forwarder is irrelevant to the application of the model. Its worth keeping a mental separation between data ingestion & distributed search components when it comes to the MLTK. If you a running a local search on the heavy forwarder, you can just move the model over as a lookup (simple knowledge object) - as long as the MLTK & PSC apps are installed. Hope this helps!

[grana_splunk · Accepted solution]

Hi,

Please find my reply below

• Is there any way to automatically or manually to export/import the model to different Splunk instances?

Assuming that you have same data and data fields in a different Splunk instance , you can copy the model file (lookup object i.e csv file) into following location (location of model files would be same in both the instances of splunk)

   <SPLUNK_HOME>/etc/apps/Splunk_ML_Toolkit/lookups

• Is there a way to share the model between Splunk and 3rd party applications (like PMML or similar)

I do not think MLTK supports sharing model between Splunk and 3rd party Applications at this moment

Note : Manually copying the model files into a different Splunk instance does not retains the same user permission