I'm running Splunk 8.2.1 in DEVTEST to learn how to use MLTK. Trial and Free licenses work, DEVTEST does not work. I left the license in trial mode, at this point I confirmed that MLTK was able to generate results for a few of the showcase items (disk space utilisation etc). Once I installed my DEVTEST license, the error is reported: Error in 'fit' command: Error while saving model "example_disk_utilization": Model does not exist. I switched the Splunk instance to Free mode, ran the example (Predict Disk Utilization) again and it began working again. Below the search.log is included. Any suggestions/advice is welcomed. Dave 08-14-2021 17:02:44.511 INFO dispatchRunner [0 MainThread] - Search process mode: freestanding (build ddff1c41e5cf). 08-14-2021 17:02:44.511 INFO dispatchRunner [0 MainThread] - initing LicenseMgr in search process: nonPro=1 08-14-2021 17:02:44.511 INFO LMStackMgr [0 MainThread] - Initializing CleMgr... 08-14-2021 17:02:44.512 INFO LicenseMgr [0 MainThread] - Initing LicenseMgr 08-14-2021 17:02:44.515 INFO ServerConfig [0 MainThread] - Found no hostname options in server.conf. Will attempt to use default for now. 08-14-2021 17:02:44.515 INFO ServerConfig [0 MainThread] - Host name option is "". 08-14-2021 17:02:44.530 INFO ServerConfig [0 MainThread] - SSL session cache path enabled 0 session timeout on SSL server 300.000 08-14-2021 17:02:44.532 INFO ServerConfig [0 MainThread] - Splunk is starting with EC-SSC disabled 08-14-2021 17:02:44.532 INFO LMConfig [0 MainThread] - serverName=homeserver guid=xxxxxxxxxxxxxxxxxxxxxxxx 08-14-2021 17:02:44.532 INFO LMConfig [0 MainThread] - connection_timeout=30 08-14-2021 17:02:44.532 INFO LMConfig [0 MainThread] - send_timeout=30 08-14-2021 17:02:44.532 INFO LMConfig [0 MainThread] - receive_timeout=30 08-14-2021 17:02:44.532 INFO LMConfig [0 MainThread] - key=license_warnings_update_interval not found in licenser stanza of server.conf, defaulting=0 08-14-2021 17:02:44.532 INFO LMConfig [0 MainThread] - squash_threshold=2000 08-14-2021 17:02:44.532 INFO LMConfig [0 MainThread] - strict_pool_quota=1 08-14-2021 17:02:44.532 INFO LMConfig [0 MainThread] - key=pool_suggestion not found in licenser stanza of server.conf, defaulting='' 08-14-2021 17:02:44.532 INFO LMConfig [0 MainThread] - key=test_aws_metering not found in licenser stanza of server.conf, defaulting=0 08-14-2021 17:02:44.532 INFO LMConfig [0 MainThread] - key=test_aws_product_code not found in licenser stanza of server.conf, defaulting=0 08-14-2021 17:02:44.532 INFO LicenseMgr [0 MainThread] - Initing LicenseMgr runContext_splunkd=false 08-14-2021 17:02:44.532 INFO LMStackMgr [0 MainThread] - closing stack mgr 08-14-2021 17:02:44.532 INFO LMSlaveInfo [0 MainThread] - all slaves cleared 08-14-2021 17:02:44.532 INFO LMStackMgr [0 MainThread] - Initalized license_warnings_update_interval=auto 08-14-2021 17:02:44.532 INFO LMStackMgr [0 MainThread] - License Manager supports Conditional Licensing Enforcement. For baked in CLE policies, window_period=60 days, max_violations=45, for stack size below 107374182400 bytes 08-14-2021 17:02:44.532 INFO LMLicense [0 MainThread] - Applying default enforcement policy for free 08-14-2021 17:02:44.532 INFO LMStackMgr [0 MainThread] - Added policy WinSz=30 Warnings=3 MaxSize=0 isDefault=1 features= for free 08-14-2021 17:02:44.532 INFO LMLicense [0 MainThread] - Applying default enforcement policy for forwarder 08-14-2021 17:02:44.532 INFO LMStackMgr [0 MainThread] - Added policy WinSz=30 Warnings=5 MaxSize=0 isDefault=1 features= for forwarder 08-14-2021 17:02:44.532 INFO LMStackMgr [0 MainThread] - Skipping trial license as alternative license type in use 08-14-2021 17:02:44.534 INFO LMStack [0 MainThread] - Added type=enterprise license, from file=Splunk (2).License.lic, to stack=enterprise of group=Enterprise 08-14-2021 17:02:44.534 INFO LMLicense [0 MainThread] - Applying default CLE policy for enterprise 08-14-2021 17:02:44.534 INFO LMStackMgr [0 MainThread] - created stack='enterprise' 08-14-2021 17:02:44.534 INFO LMStackMgr [0 MainThread] - Added policy WinSz=60 Warnings=45 MaxSize=107374182400 isDefault=1 features=LocalSearch for enterprise 08-14-2021 17:02:44.534 INFO LMStackMgr [0 MainThread] - Skipping trial pool stanza as alternative license in use 08-14-2021 17:02:44.534 INFO LMStackMgr [0 MainThread] - added pool auto_generated_pool_enterprise to stack enterprise 08-14-2021 17:02:44.534 INFO LMStackMgr [0 MainThread] - added pool auto_generated_pool_forwarder to stack forwarder 08-14-2021 17:02:44.534 INFO LMStackMgr [0 MainThread] - added pool auto_generated_pool_free to stack free 08-14-2021 17:02:44.534 INFO LMStackMgr [0 MainThread] - Initialized hideQuotaWarning = "0" 08-14-2021 17:02:44.534 INFO LMStackMgr [0 MainThread] - init completed [xxxxxxxxxxxxxxxxxxxxxx,Enterprise,runContext_splunkd=false] 08-14-2021 17:02:44.534 INFO LicenseMgr [0 MainThread] - StackMgr init complete... 08-14-2021 17:02:44.534 INFO LMTracker [0 MainThread] - Setting default product type='enterprise' 08-14-2021 17:02:44.534 INFO LMTracker [0 MainThread] - this is not splunkd, will perform partial init 08-14-2021 17:02:44.534 INFO LicenseMgr [0 MainThread] - Tracker init complete... ---- 08-14-2021 17:02:52.634 INFO DispatchExecutor [16924 phase_1] - BEGIN OPEN: Processor=inputlookup 08-14-2021 17:02:52.634 INFO DispatchExecutor [16924 phase_1] - END OPEN: Processor=inputlookup 08-14-2021 17:02:52.634 INFO DispatchExecutor [16924 phase_1] - BEGIN OPEN: Processor=fit 08-14-2021 17:02:52.688 INFO SearchOperator:inputcsv [16924 phase_1] - sid:1628931764.26 Successfully read lookup file 'C:\Program Files\Splunk\etc\apps\Splunk_ML_Toolkit\lookups\server_power.csv'. 08-14-2021 17:02:52.688 INFO PreviewExecutor [8616 StatusEnforcerThread] - Preview Enforcing initialization done 08-14-2021 17:02:52.689 INFO ReducePhaseExecutor [8616 StatusEnforcerThread] - ReducePhaseExecutor=1 action=PREVIEW 08-14-2021 17:02:52.690 INFO DispatchExecutor [16924 phase_1] - END OPEN: Processor=fit 08-14-2021 17:02:53.975 ERROR ChunkedExternProcessor [12592 ChunkedExternProcessorStderrLogger] - stderr: C:\Program Files\Splunk\etc\apps\Splunk_SA_Scientific_Python_windows_x86_64\bin\windows_x86_64\lib\site-packages\sklearn\ensemble\forest.py:245: FutureWarning: The default value of n_estimators will change from 10 in version 0.20 to 100 in 0.22. 08-14-2021 17:02:53.975 ERROR ChunkedExternProcessor [12592 ChunkedExternProcessorStderrLogger] - stderr: "10 in version 0.20 to 100 in 0.22.", FutureWarning) 08-14-2021 17:02:53.975 ERROR ChunkedExternProcessor [12592 ChunkedExternProcessorStderrLogger] - stderr: C:\Program Files\Splunk\etc\apps\Splunk_ML_Toolkit\bin\util\df_util.py:190: FutureWarning: The join_axes-keyword is deprecated. Use .reindex or .reindex_like on the result to achieve the same functionality. 08-14-2021 17:02:53.975 ERROR ChunkedExternProcessor [12592 ChunkedExternProcessorStderrLogger] - stderr: merged_df = pd.concat([original_df, additional_df], axis=1, join_axes=[original_df.index]) 08-14-2021 17:02:54.308 ERROR ChunkedExternProcessor [16924 phase_1] - Error in 'fit' command: Error while saving model "example_disk_utilization": Model does not exist 08-14-2021 17:02:54.314 INFO ReducePhaseExecutor [16924 phase_1] - Not downloading remote search.log files. Reason: No remote event providers. 08-14-2021 17:02:54.314 INFO ReducePhaseExecutor [16924 phase_1] - Not downloading remote search_telemetry.json files. Reason: No remote event providers. 08-14-2021 17:02:54.314 INFO ReducePhaseExecutor [16924 phase_1] - Ending phase_1 08-14-2021 17:02:54.314 INFO UserManager [16924 phase_1] - Unwound user context: NULL -> NULL 08-14-2021 17:02:54.314 ERROR SearchOrchestrator [7440 searchOrchestrator] - Phase_1 failed due to : Error in 'fit' command: Error while saving model "example_disk_utilization": Model does not exist 08-14-2021 17:02:54.314 INFO ReducePhaseExecutor [8616 StatusEnforcerThread] - ReducePhaseExecutor=1 action=CANCEL 08-14-2021 17:02:54.314 INFO DispatchExecutor [8616 StatusEnforcerThread] - User applied action=CANCEL while status=0 08-14-2021 17:02:54.314 ERROR SearchStatusEnforcer [8616 StatusEnforcerThread] - sid:1628931764.26 Error in 'fit' command: Error while saving model "example_disk_utilization": Model does not exist 08-14-2021 17:02:54.314 INFO SearchStatusEnforcer [8616 StatusEnforcerThread] - State changed to FAILED due to: Error in 'fit' command: Error while saving model "example_disk_utilization": Model does not exist 08-14-2021 17:02:54.332 ERROR ChunkedExternProcessor [12592 ChunkedExternProcessorStderrLogger] - stderr: Traceback (most recent call last): 08-14-2021 17:02:54.332 ERROR ChunkedExternProcessor [12592 ChunkedExternProcessorStderrLogger] - stderr: File "C:\Program Files\Splunk\etc\apps\Splunk_ML_Toolkit\bin\util\lookups_util.py", line 73, in parse_model_reply 08-14-2021 17:02:54.332 ERROR ChunkedExternProcessor [12592 ChunkedExternProcessorStderrLogger] - stderr: return lookups_parse_reply(reply) 08-14-2021 17:02:54.332 ERROR ChunkedExternProcessor [12592 ChunkedExternProcessorStderrLogger] - stderr: File "C:\Program Files\Splunk\etc\apps\Splunk_ML_Toolkit\bin\util\lookups_util.py", line 57, in lookups_parse_reply 08-14-2021 17:02:54.332 ERROR ChunkedExternProcessor [12592 ChunkedExternProcessorStderrLogger] - stderr: raise e 08-14-2021 17:02:54.332 ERROR ChunkedExternProcessor [12592 ChunkedExternProcessorStderrLogger] - stderr: File "C:\Program Files\Splunk\etc\apps\Splunk_ML_Toolkit\bin\util\lookups_util.py", line 47, in lookups_parse_reply 08-14-2021 17:02:54.332 ERROR ChunkedExternProcessor [12592 ChunkedExternProcessorStderrLogger] - stderr: raise LookupNotFoundException() 08-14-2021 17:02:54.332 ERROR ChunkedExternProcessor [12592 ChunkedExternProcessorStderrLogger] - stderr: util.lookup_exceptions.LookupNotFoundException: Lookup does not exist 08-14-2021 17:02:54.332 ERROR ChunkedExternProcessor [12592 ChunkedExternProcessorStderrLogger] - stderr: During handling of the above exception, another exception occurred: 08-14-2021 17:02:54.332 ERROR ChunkedExternProcessor [12592 ChunkedExternProcessorStderrLogger] - stderr: Traceback (most recent call last): 08-14-2021 17:02:54.332 ERROR ChunkedExternProcessor [12592 ChunkedExternProcessorStderrLogger] - stderr: File "C:\Program Files\Splunk\etc\apps\Splunk_ML_Toolkit\bin\processors\FitBatchProcessor.py", line 202, in save_model 08-14-2021 17:02:54.332 ERROR ChunkedExternProcessor [12592 ChunkedExternProcessorStderrLogger] - stderr: namespace=self.namespace, 08-14-2021 17:02:54.332 ERROR ChunkedExternProcessor [12592 ChunkedExternProcessorStderrLogger] - stderr: File "C:\Program Files\Splunk\etc\apps\Splunk_ML_Toolkit\bin\models\base.py", line 177, in save_model 08-14-2021 17:02:54.332 ERROR ChunkedExternProcessor [12592 ChunkedExternProcessorStderrLogger] - stderr: parse_model_reply(reply) 08-14-2021 17:02:54.332 ERROR ChunkedExternProcessor [12592 ChunkedExternProcessorStderrLogger] - stderr: File "C:\Program Files\Splunk\etc\apps\Splunk_ML_Toolkit\bin\util\lookups_util.py", line 75, in parse_model_reply 08-14-2021 17:02:54.332 ERROR ChunkedExternProcessor [12592 ChunkedExternProcessorStderrLogger] - stderr: raise ModelNotFoundException() 08-14-2021 17:02:54.332 ERROR ChunkedExternProcessor [12592 ChunkedExternProcessorStderrLogger] - stderr: util.lookup_exceptions.ModelNotFoundException: Model does not exist 08-14-2021 17:02:54.332 ERROR ChunkedExternProcessor [12592 ChunkedExternProcessorStderrLogger] - stderr: Error while saving model "example_disk_utilization": Model does not exist 08-14-2021 17:02:54.332 ERROR ChunkedExternProcessor [12592 ChunkedExternProcessorStderrLogger] - stderr: Traceback (most recent call last): 08-14-2021 17:02:54.332 ERROR ChunkedExternProcessor [12592 ChunkedExternProcessorStderrLogger] - stderr: File "C:\Program Files\Splunk\etc\apps\Splunk_ML_Toolkit\bin\util\lookups_util.py", line 73, in parse_model_reply 08-14-2021 17:02:54.332 ERROR ChunkedExternProcessor [12592 ChunkedExternProcessorStderrLogger] - stderr: return lookups_parse_reply(reply) 08-14-2021 17:02:54.332 ERROR ChunkedExternProcessor [12592 ChunkedExternProcessorStderrLogger] - stderr: File "C:\Program Files\Splunk\etc\apps\Splunk_ML_Toolkit\bin\util\lookups_util.py", line 57, in lookups_parse_reply 08-14-2021 17:02:54.332 ERROR ChunkedExternProcessor [12592 ChunkedExternProcessorStderrLogger] - stderr: raise e 08-14-2021 17:02:54.332 ERROR ChunkedExternProcessor [12592 ChunkedExternProcessorStderrLogger] - stderr: File "C:\Program Files\Splunk\etc\apps\Splunk_ML_Toolkit\bin\util\lookups_util.py", line 47, in lookups_parse_reply 08-14-2021 17:02:54.332 ERROR ChunkedExternProcessor [12592 ChunkedExternProcessorStderrLogger] - stderr: raise LookupNotFoundException() 08-14-2021 17:02:54.332 ERROR ChunkedExternProcessor [12592 ChunkedExternProcessorStderrLogger] - stderr: util.lookup_exceptions.LookupNotFoundException: Lookup does not exist 08-14-2021 17:02:54.332 ERROR ChunkedExternProcessor [12592 ChunkedExternProcessorStderrLogger] - stderr: During handling of the above exception, another exception occurred: 08-14-2021 17:02:54.332 ERROR ChunkedExternProcessor [12592 ChunkedExternProcessorStderrLogger] - stderr: Traceback (most recent call last): 08-14-2021 17:02:54.332 ERROR ChunkedExternProcessor [12592 ChunkedExternProcessorStderrLogger] - stderr: File "C:\Program Files\Splunk\etc\apps\Splunk_ML_Toolkit\bin\processors\FitBatchProcessor.py", line 202, in save_model 08-14-2021 17:02:54.332 ERROR ChunkedExternProcessor [12592 ChunkedExternProcessorStderrLogger] - stderr: namespace=self.namespace, 08-14-2021 17:02:54.332 ERROR ChunkedExternProcessor [12592 ChunkedExternProcessorStderrLogger] - stderr: File "C:\Program Files\Splunk\etc\apps\Splunk_ML_Toolkit\bin\models\base.py", line 177, in save_model 08-14-2021 17:02:54.332 ERROR ChunkedExternProcessor [12592 ChunkedExternProcessorStderrLogger] - stderr: parse_model_reply(reply) 08-14-2021 17:02:54.332 ERROR ChunkedExternProcessor [12592 ChunkedExternProcessorStderrLogger] - stderr: File "C:\Program Files\Splunk\etc\apps\Splunk_ML_Toolkit\bin\util\lookups_util.py", line 75, in parse_model_reply 08-14-2021 17:02:54.332 ERROR ChunkedExternProcessor [12592 ChunkedExternProcessorStderrLogger] - stderr: raise ModelNotFoundException() 08-14-2021 17:02:54.332 ERROR ChunkedExternProcessor [12592 ChunkedExternProcessorStderrLogger] - stderr: util.lookup_exceptions.ModelNotFoundException: Model does not exist 08-14-2021 17:02:54.332 ERROR ChunkedExternProcessor [12592 ChunkedExternProcessorStderrLogger] - stderr: During handling of the above exception, another exception occurred: 08-14-2021 17:02:54.332 ERROR ChunkedExternProcessor [12592 ChunkedExternProcessorStderrLogger] - stderr: Traceback (most recent call last): 08-14-2021 17:02:54.332 ERROR ChunkedExternProcessor [12592 ChunkedExternProcessorStderrLogger] - stderr: File "C:\Program Files\Splunk\etc\apps\Splunk_ML_Toolkit\bin\cexc\__init__.py", line 174, in run 08-14-2021 17:02:54.332 ERROR ChunkedExternProcessor [12592 ChunkedExternProcessorStderrLogger] - stderr: while self._handle_chunk(): 08-14-2021 17:02:54.332 ERROR ChunkedExternProcessor [12592 ChunkedExternProcessorStderrLogger] - stderr: File "C:\Program Files\Splunk\etc\apps\Splunk_ML_Toolkit\bin\cexc\__init__.py", line 236, in _handle_chunk 08-14-2021 17:02:54.332 ERROR ChunkedExternProcessor [12592 ChunkedExternProcessorStderrLogger] - stderr: ret = self.handler(metadata, body) 08-14-2021 17:02:54.332 ERROR ChunkedExternProcessor [12592 ChunkedExternProcessorStderrLogger] - stderr: File "C:\Program Files\Splunk\etc\apps\Splunk_ML_Toolkit\bin\fit.py", line 154, in handler 08-14-2021 17:02:54.332 ERROR ChunkedExternProcessor [12592 ChunkedExternProcessorStderrLogger] - stderr: self.controller.finalize() 08-14-2021 17:02:54.332 ERROR ChunkedExternProcessor [12592 ChunkedExternProcessorStderrLogger] - stderr: File "C:\Program Files\Splunk\etc\apps\Splunk_ML_Toolkit\bin\chunked_controller.py", line 241, in finalize 08-14-2021 17:02:54.332 ERROR ChunkedExternProcessor [12592 ChunkedExternProcessorStderrLogger] - stderr: self.processor.save_model() 08-14-2021 17:02:54.332 ERROR ChunkedExternProcessor [12592 ChunkedExternProcessorStderrLogger] - stderr: File "C:\Program Files\Splunk\etc\apps\Splunk_ML_Toolkit\bin\processors\FitBatchProcessor.py", line 207, in save_model 08-14-2021 17:02:54.332 ERROR ChunkedExternProcessor [12592 ChunkedExternProcessorStderrLogger] - stderr: 'Error while saving model "%s": %s' % (self.algo_options['model_name'], e) 08-14-2021 17:02:54.333 ERROR ChunkedExternProcessor [12592 ChunkedExternProcessorStderrLogger] - stderr: RuntimeError: Error while saving model "example_disk_utilization": Model does not exist 08-14-2021 17:02:54.343 INFO UserManager [8616 StatusEnforcerThread] - Unwound user context: NULL -> NULL 08-14-2021 17:02:54.345 INFO DispatchStorageManager [7440 searchOrchestrator] - Remote storage disabled for search artifacts. 08-14-2021 17:02:54.345 INFO DispatchManager [7440 searchOrchestrator] - DispatchManager::dispatchHasFinished(id='1628931764.26', username='admin') 08-14-2021 17:02:54.346 INFO UserManager [7440 searchOrchestrator] - Unwound user context: NULL -> NULL 08-14-2021 17:02:54.410 INFO SearchStatusEnforcer [13600 RunDispatch] - SearchStatusEnforcer is already terminated 08-14-2021 17:02:54.410 INFO UserManager [13600 RunDispatch] - Unwound user context: NULL -> NULL 08-14-2021 17:02:54.410 INFO LookupDataProvider [13600 RunDispatch] - Clearing out lookup shared provider map 08-14-2021 17:02:54.436 INFO ShutdownHandler [15220 Shutdown] - Shutting down splunkd ---- 08-14-2021 17:02:54.436 INFO ShutdownHandler [15220 Shutdown] - Shutdown complete in 0 microseconds 08-14-2021 17:02:54.438 INFO HealthReporter [15936 MainThread] - aggregate_ingestion_latency_health with value=0 from stanza=health_reporter will disable the aggregation of ingestion latency health reporter. 08-14-2021 17:02:54.438 ERROR dispatchRunner [15936 MainThread] - RunDispatch::runDispatchThread threw error: Error in 'fit' command: Error while saving model "example_disk_utilization": Model does not exist
... View more