All Apps and Add-ons

How to resolve Splunk machine learning knowledge bundle size issues?

Path Finder

We run a number of machine learning models and routinely run into limitations of the "knowledge bundle" getting too big with errors like this

bundle errorsbundle errors


We increased the limits.conf to alleviate it but error came back after a few more models were made.

I've noticed that these likely need to be included in the knowledge bundle since they are not explicitly blacklisted from the distsearch.conf





replicate.algos = true
replicate.mlspl = true
replicate.scorings = true

non_model_lookups = apps[/\\]Splunk_ML_Toolkit[/\\]lookups[/\\](?!__mlspl_)*.csv
non_model_lookups_docs = apps[/\\]Splunk_ML_Toolkit[/\\]lookups[/\\]docs[/\\]...





Now looking at the users directory there are a lot of double ups. 


users ML lookup directoryusers ML lookup directory

Is there a way to get rid of these _draft_ ones in the Machine Learning GUI?


Labels (1)
0 Karma

Path Finder

Answering my own question here. I did some testing and the _draft_ ones are updated when you are playing around in the experiments section with each model types interfaces.

They update the non _draft_ ones when you save. Which is pretty obvious.

As for whether the models are required in the knowledge bundle it all depends on the search you are running to build your data set. If the job inspector phase0 search includes the fit command then you will likely run into issues if you don't distribute these in the bundle.

I added this to /opt/splunk/etc/apps/Splunk_ML_Toolkit/local/distsearch.conf to see if I could "break" it.

# non_draft_models = apps/Splunk_ML_Toolkit/lookups/(?!__mlspl__exp_draft)*.mlmodel
user_models = users/*/Splunk_ML_Toolkit/lookups/*.mlmodel
# all_models = apps/Splunk_ML_Toolkit/lookups/*.mlmodel


Essentially stops all users data models being included in the knowledge bundle.

I tried many different combinations of searches using fit and couldn't get a search that tried to run the fit command on the indexers. Like this pointless model

index=_internal (status=2* OR status=3* OR status=4*)
| fit DensityFunction "status" by "method" dist=auto threshold=0.0001 show_density=true show_options="feature_variables,split_by,params" into "_exp_draft_212803036b7b408a9d467eedf8c6a3a6"


The fit always ran on the search head so I'd say 9 times out of 10 you don't need these in the bundle.

0 Karma

Path Finder

Replying to my own post of the other requirement if you have large knowledge bundles. You need this on the indexers server.conf as well if you continue to have bundles bigger than 2GB.



# Required on indexers due to increasing knowledge bundle sizes to 5GB. Large Knowledge Bundles caused by machine learning models. Default is 2147483648 (2GB)
max_content_length = 5368709120


0 Karma
Get Updates on the Splunk Community!

Dashboard Studio Challenge - Learn New Tricks, Showcase Your Skills, and Win Prizes!

Reimagine what you can do with your dashboards. Dashboard Studio is Splunk’s newest dashboard builder to ...

Introducing Edge Processor: Next Gen Data Transformation

We get it - not only can it take a lot of time, money and resources to get data into Splunk, but it also takes ...

Take the 2021 Splunk Career Survey for $50 in Amazon Cash

Help us learn about how Splunk has impacted your career by taking the 2021 Splunk Career Survey. Last year’s ...