All Apps and Add-ons

MLTK: How best to conditionally update model in another app based on R squared value

burwell
SplunkTrust
SplunkTrust

I am starting to use MLTK 4.1.0 with Random Forest Regressor.

I understand how to use the experiments and then publish a model into another app.
I understand how to schedule model training.

What I was wondering is how to validate the new mode before publishing into another app. I can see I can fit into app:model_name. But I don't want to update my app unless r2_score > 0.90 some value.

I know I can run the score command after the apply to derive the r2_score but it seems kludgy at best on how I would then rerun the fit to fit into the model in another app.

Maybe I am approaching the workflow for MLTK all wrong.

1 Solution

grana_splunk
Splunk Employee
Splunk Employee

Nice question!!

Currently its not available out of box in MLTK but can be done using some alerts and custom scripts. Just make sure the user has the right permission and role in Splunk. Its a logical ask I must say.

Regards,

View solution in original post

0 Karma

grana_splunk
Splunk Employee
Splunk Employee

Nice question!!

Currently its not available out of box in MLTK but can be done using some alerts and custom scripts. Just make sure the user has the right permission and role in Splunk. Its a logical ask I must say.

Regards,

View solution in original post

0 Karma

burwell
SplunkTrust
SplunkTrust

So @grana_splunk the solution would involve alerting on the r2_score which could run a custom script that would do the publishing? Looking at the value and then setting a field and using custom actions.

Would the script merely copy the model from the mltk to my other app?

So something like

cp /opt/splunk/etc/<myusername>/Splunk_ML_Toolkit/lookups/__mlspl__<experiment_number>.csv /opt/splunk/etc/users/<myusername>/<myapp>/lookups/__mlspl_<published_name_of_model>.csv

OR if my model was public I would do this copy

cp /opt/splunk/etc//Splunk_ML_Toolkit/lookups/mlsplexp_2f88d61ca98745a586df36aeabd78dcb.csv /opt/splunk/etc/apps//lookups/mlspl.csv

0 Karma

grana_splunk
Splunk Employee
Splunk Employee

Ensure the app namespace and user namespace is correct and please check the roles and permissions. for now alerting and then copying will do the job for you.

0 Karma

burwell
SplunkTrust
SplunkTrust

Thanks.

So I submitted a feature request since there is no way to do this from MLTK. Seems like having more automation around model validation is needed.

Also, for us, I didn't want to do the script/copying. Instead I am going to manually publish for now.

0 Karma

grana_splunk
Splunk Employee
Splunk Employee

Great thanks.

0 Karma

skoelpin
SplunkTrust
SplunkTrust

What kind've problem are you trying to solve? Would an offline model do the trick? How often are you training?

0 Karma

burwell
SplunkTrust
SplunkTrust

So I am trying use the experiments feature, if the my r2_score is > 0.9 then I publish to another app.

0 Karma

skoelpin
SplunkTrust
SplunkTrust

I think I got a working process for this.. I just tested it and it works!

Go to your experiment tab, run your last model, and pop open the search that calculates the sSquared value and RMSE. Append the SPL below on the search

| eval ALERT=if(rSquared>0.94,"PASS","FAIL")

Add the splunk_search alert action onto the search which will house the fit step. Set the alert action to run if ALERT="PASS". This will ensure only runs with an rSquared value above 95% will be applied to the model

0 Karma

burwell
SplunkTrust
SplunkTrust

Thanks. You are referring to this app https://splunkbase.splunk.com/app/3837/ which I don't have installed at this time.

Thanks.

0 Karma
Did you miss .conf21 Virtual?

Good news! The event's keynotes and many of its breakout sessions are now available online, and still totally FREE!