All Apps and Add-ons

How to resolve Splunk machine learning knowledge bundle size issues?

cameronjust
Path Finder

We run a number of machine learning models and routinely run into limitations of the "knowledge bundle" getting too big with errors like this

bundle errorsbundle errors

 

We increased the limits.conf to alleviate it but error came back after a few more models were made.

I've noticed that these likely need to be included in the knowledge bundle since they are not explicitly blacklisted from the distsearch.conf

 

 

 

 

[replicationSettings:refineConf]
replicate.algos = true
replicate.mlspl = true
replicate.scorings = true

[replicationBlacklist]
non_model_lookups = apps[/\\]Splunk_ML_Toolkit[/\\]lookups[/\\](?!__mlspl_)*.csv
non_model_lookups_docs = apps[/\\]Splunk_ML_Toolkit[/\\]lookups[/\\]docs[/\\]...

 

 

 

 

Now looking at the users directory there are a lot of double ups. 

/opt/splunk/etc/users/theusername/Splunk_ML_Toolkit/lookups

users ML lookup directoryusers ML lookup directory

Is there a way to get rid of these _draft_ ones in the Machine Learning GUI?

 

Labels (1)
0 Karma

cameronjust
Path Finder

Answering my own question here. I did some testing and the _draft_ ones are updated when you are playing around in the experiments section with each model types interfaces.

They update the non _draft_ ones when you save. Which is pretty obvious.

As for whether the models are required in the knowledge bundle it all depends on the search you are running to build your data set. If the job inspector phase0 search includes the fit command then you will likely run into issues if you don't distribute these in the bundle.

I added this to /opt/splunk/etc/apps/Splunk_ML_Toolkit/local/distsearch.conf to see if I could "break" it.

[replicationBlacklist]
# non_draft_models = apps/Splunk_ML_Toolkit/lookups/(?!__mlspl__exp_draft)*.mlmodel
user_models = users/*/Splunk_ML_Toolkit/lookups/*.mlmodel
# all_models = apps/Splunk_ML_Toolkit/lookups/*.mlmodel

 

Essentially stops all users data models being included in the knowledge bundle.

I tried many different combinations of searches using fit and couldn't get a search that tried to run the fit command on the indexers. Like this pointless model

index=_internal (status=2* OR status=3* OR status=4*)
| fit DensityFunction "status" by "method" dist=auto threshold=0.0001 show_density=true show_options="feature_variables,split_by,params" into "_exp_draft_212803036b7b408a9d467eedf8c6a3a6"

 

The fit always ran on the search head so I'd say 9 times out of 10 you don't need these in the bundle.

0 Karma

cameronjust
Path Finder

Replying to my own post of the other requirement if you have large knowledge bundles. You need this on the indexers server.conf as well if you continue to have bundles bigger than 2GB.

 

[httpServer]

# Required on indexers due to increasing knowledge bundle sizes to 5GB. Large Knowledge Bundles caused by machine learning models. Default is 2147483648 (2GB)
max_content_length = 5368709120

 

0 Karma
Get Updates on the Splunk Community!

Splunk Training for All: Meet Aspiring Cybersecurity Analyst, Marc Alicea

Splunk Education believes in the value of training and certification in today’s rapidly-changing data-driven ...

Investigate Security and Threat Detection with VirusTotal and Splunk Integration

As security threats and their complexities surge, security analysts deal with increased challenges and ...

Observability Highlights | January 2023 Newsletter

 January 2023New Product Releases Splunk Network Explorer for Infrastructure MonitoringSplunk unveils Network ...