topic How to resolve Splunk machine learning knowledge bundle size issues? in All Apps and Add-ons

How to resolve Splunk machine learning knowledge bundle size issues?

cameronjust — Wed, 25 Jan 2023 15:11:35 GMT

We run a number of machine learning models and routinely run into limitations of the "knowledge bundle" getting too big with errors like this

bundle errors

We increased the limits.conf to alleviate it but error came back after a few more models were made.

I've noticed that these likely need to be included in the knowledge bundle since they are not explicitly blacklisted from the distsearch.conf

[replicationSettings:refineConf] replicate.algos = true replicate.mlspl = true replicate.scorings = true [replicationBlacklist] non_model_lookups = apps[/\\]Splunk_ML_Toolkit[/\\]lookups[/\\](?!__mlspl_)*.csv non_model_lookups_docs = apps[/\\]Splunk_ML_Toolkit[/\\]lookups[/\\]docs[/\\]...

Now looking at the users directory there are a lot of double ups.

/opt/splunk/etc/users/theusername/Splunk_ML_Toolkit/lookups

users ML lookup directory

Is there a way to get rid of these _draft_ ones in the Machine Learning GUI?

Re: How to Resolve Splunk Machine Learning Knowledge Bundle Size Issues?

cameronjust — Thu, 25 Aug 2022 23:27:07 GMT

Answering my own question here. I did some testing and the _draft_ ones are updated when you are playing around in the experiments section with each model types interfaces.

They update the non _draft_ ones when you save. Which is pretty obvious.

As for whether the models are required in the knowledge bundle it all depends on the search you are running to build your data set. If the job inspector phase0 search includes the fit command then you will likely run into issues if you don't distribute these in the bundle.

I added this to /opt/splunk/etc/apps/Splunk_ML_Toolkit/local/distsearch.conf to see if I could "break" it.

[replicationBlacklist] # non_draft_models = apps/Splunk_ML_Toolkit/lookups/(?!__mlspl__exp_draft)*.mlmodel user_models = users/*/Splunk_ML_Toolkit/lookups/*.mlmodel # all_models = apps/Splunk_ML_Toolkit/lookups/*.mlmodel

Essentially stops all users data models being included in the knowledge bundle.

I tried many different combinations of searches using fit and couldn't get a search that tried to run the fit command on the indexers. Like this pointless model

index=_internal (status=2* OR status=3* OR status=4*) | fit DensityFunction "status" by "method" dist=auto threshold=0.0001 show_density=true show_options="feature_variables,split_by,params" into "_exp_draft_212803036b7b408a9d467eedf8c6a3a6"

The fit always ran on the search head so I'd say 9 times out of 10 you don't need these in the bundle.

Re: How to Resolve Splunk Machine Learning Knowledge Bundle Size Issues?

cameronjust — Wed, 25 Jan 2023 06:19:43 GMT

Replying to my own post of the other requirement if you have large knowledge bundles. You need this on the indexers server.conf as well if you continue to have bundles bigger than 2GB.

[httpServer] # Required on indexers due to increasing knowledge bundle sizes to 5GB. Large Knowledge Bundles caused by machine learning models. Default is 2147483648 (2GB) max_content_length = 5368709120