MLTK: How best to conditionally update model in another app based on R squared value

[burwell]

I am starting to use MLTK 4.1.0 with Random Forest Regressor.

I understand how to use the experiments and then publish a model into another app.
I understand how to schedule model training.

What I was wondering is how to validate the new mode before publishing into another app. I can see I can fit into app:model_name. But I don't want to update my app unless r2_score > 0.90 some value.

I know I can run the score command after the apply to derive the r2_score but it seems kludgy at best on how I would then rerun the fit to fit into the model in another app.

Maybe I am approaching the workflow for MLTK all wrong.

[grana_splunk · Accepted solution]

Nice question!!

Currently its not available out of box in MLTK but can be done using some alerts and custom scripts. Just make sure the user has the right permission and role in Splunk. Its a logical ask I must say.

Regards,

[burwell]

So @grana_splunk the solution would involve alerting on the r2_score which could run a custom script that would do the publishing? Looking at the value and then setting a field and using custom actions.

Would the script merely copy the model from the mltk to my other app?

So something like

cp /opt/splunk/etc/<myusername>/Splunk_ML_Toolkit/lookups/__mlspl__<experiment_number>.csv /opt/splunk/etc/users/<myusername>/<myapp>/lookups/__mlspl_<published_name_of_model>.csv

OR if my model was public I would do this copy

cp /opt/splunk/etc//Splunk_ML_Toolkit/lookups/mlsplexp_2f88d61ca98745a586df36aeabd78dcb.csv /opt/splunk/etc/apps//lookups/mlspl.csv

[grana_splunk]

Ensure the app namespace and user namespace is correct and please check the roles and permissions. for now alerting and then copying will do the job for you.

[burwell]

Thanks.

So I submitted a feature request since there is no way to do this from MLTK. Seems like having more automation around model validation is needed.

Also, for us, I didn't want to do the script/copying. Instead I am going to manually publish for now.

[grana_splunk]

Great thanks.

[skoelpin]

What kind've problem are you trying to solve? Would an offline model do the trick? How often are you training?

[burwell]

So I am trying use the experiments feature, if the my r2_score is > 0.9 then I publish to another app.

[skoelpin]

I think I got a working process for this.. I just tested it and it works!

Go to your experiment tab, run your last model, and pop open the search that calculates the sSquared value and RMSE. Append the SPL below on the search

| eval ALERT=if(rSquared>0.94,"PASS","FAIL")

Add the splunk_search alert action onto the search which will house the fit step. Set the alert action to run if ALERT="PASS". This will ensure only runs with an rSquared value above 95% will be applied to the model

[burwell]

Thanks. You are referring to this app https://splunkbase.splunk.com/app/3837/ which I don't have installed at this time.

Thanks.