How to resolve Splunk machine learning knowledge bundle size issues?

[cameronjust]

We run a number of machine learning models and routinely run into limitations of the "knowledge bundle" getting too big with errors like this

bundle errors

We increased the limits.conf to alleviate it but error came back after a few more models were made.

I've noticed that these likely need to be included in the knowledge bundle since they are not explicitly blacklisted from the distsearch.conf

[replicationSettings:refineConf]
replicate.algos = true
replicate.mlspl = true
replicate.scorings = true

[replicationBlacklist]
non_model_lookups = apps[/\\]Splunk_ML_Toolkit[/\\]lookups[/\\](?!__mlspl_)*.csv
non_model_lookups_docs = apps[/\\]Splunk_ML_Toolkit[/\\]lookups[/\\]docs[/\\]...

Now looking at the users directory there are a lot of double ups. 

/opt/splunk/etc/users/theusername/Splunk_ML_Toolkit/lookups

users ML lookup directory

Is there a way to get rid of these _draft_ ones in the Machine Learning GUI?

[cameronjust]

Answering my own question here. I did some testing and the _draft_ ones are updated when you are playing around in the experiments section with each model types interfaces.

They update the non _draft_ ones when you save. Which is pretty obvious.

As for whether the models are required in the knowledge bundle it all depends on the search you are running to build your data set. If the job inspector phase0 search includes the fit command then you will likely run into issues if you don't distribute these in the bundle.

I added this to /opt/splunk/etc/apps/Splunk_ML_Toolkit/local/distsearch.conf to see if I could "break" it.

[replicationBlacklist]
# non_draft_models = apps/Splunk_ML_Toolkit/lookups/(?!__mlspl__exp_draft)*.mlmodel
user_models = users/*/Splunk_ML_Toolkit/lookups/*.mlmodel
# all_models = apps/Splunk_ML_Toolkit/lookups/*.mlmodel

Essentially stops all users data models being included in the knowledge bundle.

I tried many different combinations of searches using fit and couldn't get a search that tried to run the fit command on the indexers. Like this pointless model

index=_internal (status=2* OR status=3* OR status=4*)
| fit DensityFunction "status" by "method" dist=auto threshold=0.0001 show_density=true show_options="feature_variables,split_by,params" into "_exp_draft_212803036b7b408a9d467eedf8c6a3a6"

The fit always ran on the search head so I'd say 9 times out of 10 you don't need these in the bundle.

[cameronjust]

Replying to my own post of the other requirement if you have large knowledge bundles. You need this on the indexers server.conf as well if you continue to have bundles bigger than 2GB.

[httpServer]

# Required on indexers due to increasing knowledge bundle sizes to 5GB. Large Knowledge Bundles caused by machine learning models. Default is 2147483648 (2GB)
max_content_length = 5368709120