I'm running Splunk 8.2.1 in DEVTEST to learn how to use MLTK. Trial and Free licenses work, DEVTEST does not work.
I left the license in trial mode, at this point I confirmed that MLTK was able to generate results for a few of the showcase items (disk space utilisation etc). Once I installed my DEVTEST license, the error is reported:
Error in 'fit' command: Error while saving model "example_disk_utilization": Model does not exist.
I switched the Splunk instance to Free mode, ran the example (Predict Disk Utilization) again and it began working again.
Below the search.log is included. Any suggestions/advice is welcomed.
Dave
08-14-2021 17:02:44.511 INFO dispatchRunner [0 MainThread] - Search process mode: freestanding (build ddff1c41e5cf).
08-14-2021 17:02:44.511 INFO dispatchRunner [0 MainThread] - initing LicenseMgr in search process: nonPro=1
08-14-2021 17:02:44.511 INFO LMStackMgr [0 MainThread] - Initializing CleMgr...
08-14-2021 17:02:44.512 INFO LicenseMgr [0 MainThread] - Initing LicenseMgr
08-14-2021 17:02:44.515 INFO ServerConfig [0 MainThread] - Found no hostname options in server.conf. Will attempt to use default for now.
08-14-2021 17:02:44.515 INFO ServerConfig [0 MainThread] - Host name option is "".
08-14-2021 17:02:44.530 INFO ServerConfig [0 MainThread] - SSL session cache path enabled 0 session timeout on SSL server 300.000
08-14-2021 17:02:44.532 INFO ServerConfig [0 MainThread] - Splunk is starting with EC-SSC disabled
08-14-2021 17:02:44.532 INFO LMConfig [0 MainThread] - serverName=homeserver guid=xxxxxxxxxxxxxxxxxxxxxxxx
08-14-2021 17:02:44.532 INFO LMConfig [0 MainThread] - connection_timeout=30
08-14-2021 17:02:44.532 INFO LMConfig [0 MainThread] - send_timeout=30
08-14-2021 17:02:44.532 INFO LMConfig [0 MainThread] - receive_timeout=30
08-14-2021 17:02:44.532 INFO LMConfig [0 MainThread] - key=license_warnings_update_interval not found in licenser stanza of server.conf, defaulting=0
08-14-2021 17:02:44.532 INFO LMConfig [0 MainThread] - squash_threshold=2000
08-14-2021 17:02:44.532 INFO LMConfig [0 MainThread] - strict_pool_quota=1
08-14-2021 17:02:44.532 INFO LMConfig [0 MainThread] - key=pool_suggestion not found in licenser stanza of server.conf, defaulting=''
08-14-2021 17:02:44.532 INFO LMConfig [0 MainThread] - key=test_aws_metering not found in licenser stanza of server.conf, defaulting=0
08-14-2021 17:02:44.532 INFO LMConfig [0 MainThread] - key=test_aws_product_code not found in licenser stanza of server.conf, defaulting=0
08-14-2021 17:02:44.532 INFO LicenseMgr [0 MainThread] - Initing LicenseMgr runContext_splunkd=false
08-14-2021 17:02:44.532 INFO LMStackMgr [0 MainThread] - closing stack mgr
08-14-2021 17:02:44.532 INFO LMSlaveInfo [0 MainThread] - all slaves cleared
08-14-2021 17:02:44.532 INFO LMStackMgr [0 MainThread] - Initalized license_warnings_update_interval=auto
08-14-2021 17:02:44.532 INFO LMStackMgr [0 MainThread] - License Manager supports Conditional Licensing Enforcement. For baked in CLE policies, window_period=60 days, max_violations=45, for stack size below 107374182400 bytes
08-14-2021 17:02:44.532 INFO LMLicense [0 MainThread] - Applying default enforcement policy for free
08-14-2021 17:02:44.532 INFO LMStackMgr [0 MainThread] - Added policy WinSz=30 Warnings=3 MaxSize=0 isDefault=1 features= for free
08-14-2021 17:02:44.532 INFO LMLicense [0 MainThread] - Applying default enforcement policy for forwarder
08-14-2021 17:02:44.532 INFO LMStackMgr [0 MainThread] - Added policy WinSz=30 Warnings=5 MaxSize=0 isDefault=1 features= for forwarder
08-14-2021 17:02:44.532 INFO LMStackMgr [0 MainThread] - Skipping trial license as alternative license type in use
08-14-2021 17:02:44.534 INFO LMStack [0 MainThread] - Added type=enterprise license, from file=Splunk (2).License.lic, to stack=enterprise of group=Enterprise
08-14-2021 17:02:44.534 INFO LMLicense [0 MainThread] - Applying default CLE policy for enterprise
08-14-2021 17:02:44.534 INFO LMStackMgr [0 MainThread] - created stack='enterprise'
08-14-2021 17:02:44.534 INFO LMStackMgr [0 MainThread] - Added policy WinSz=60 Warnings=45 MaxSize=107374182400 isDefault=1 features=LocalSearch for enterprise
08-14-2021 17:02:44.534 INFO LMStackMgr [0 MainThread] - Skipping trial pool stanza as alternative license in use
08-14-2021 17:02:44.534 INFO LMStackMgr [0 MainThread] - added pool auto_generated_pool_enterprise to stack enterprise
08-14-2021 17:02:44.534 INFO LMStackMgr [0 MainThread] - added pool auto_generated_pool_forwarder to stack forwarder
08-14-2021 17:02:44.534 INFO LMStackMgr [0 MainThread] - added pool auto_generated_pool_free to stack free
08-14-2021 17:02:44.534 INFO LMStackMgr [0 MainThread] - Initialized hideQuotaWarning = "0"
08-14-2021 17:02:44.534 INFO LMStackMgr [0 MainThread] - init completed [xxxxxxxxxxxxxxxxxxxxxx,Enterprise,runContext_splunkd=false]
08-14-2021 17:02:44.534 INFO LicenseMgr [0 MainThread] - StackMgr init complete...
08-14-2021 17:02:44.534 INFO LMTracker [0 MainThread] - Setting default product type='enterprise'
08-14-2021 17:02:44.534 INFO LMTracker [0 MainThread] - this is not splunkd, will perform partial init
08-14-2021 17:02:44.534 INFO LicenseMgr [0 MainThread] - Tracker init complete...
----
08-14-2021 17:02:52.634 INFO DispatchExecutor [16924 phase_1] - BEGIN OPEN: Processor=inputlookup
08-14-2021 17:02:52.634 INFO DispatchExecutor [16924 phase_1] - END OPEN: Processor=inputlookup
08-14-2021 17:02:52.634 INFO DispatchExecutor [16924 phase_1] - BEGIN OPEN: Processor=fit
08-14-2021 17:02:52.688 INFO SearchOperator:inputcsv [16924 phase_1] - sid:1628931764.26 Successfully read lookup file 'C:\Program Files\Splunk\etc\apps\Splunk_ML_Toolkit\lookups\server_power.csv'.
08-14-2021 17:02:52.688 INFO PreviewExecutor [8616 StatusEnforcerThread] - Preview Enforcing initialization done
08-14-2021 17:02:52.689 INFO ReducePhaseExecutor [8616 StatusEnforcerThread] - ReducePhaseExecutor=1 action=PREVIEW
08-14-2021 17:02:52.690 INFO DispatchExecutor [16924 phase_1] - END OPEN: Processor=fit
08-14-2021 17:02:53.975 ERROR ChunkedExternProcessor [12592 ChunkedExternProcessorStderrLogger] - stderr: C:\Program Files\Splunk\etc\apps\Splunk_SA_Scientific_Python_windows_x86_64\bin\windows_x86_64\lib\site-packages\sklearn\ensemble\forest.py:245: FutureWarning: The default value of n_estimators will change from 10 in version 0.20 to 100 in 0.22.
08-14-2021 17:02:53.975 ERROR ChunkedExternProcessor [12592 ChunkedExternProcessorStderrLogger] - stderr: "10 in version 0.20 to 100 in 0.22.", FutureWarning)
08-14-2021 17:02:53.975 ERROR ChunkedExternProcessor [12592 ChunkedExternProcessorStderrLogger] - stderr: C:\Program Files\Splunk\etc\apps\Splunk_ML_Toolkit\bin\util\df_util.py:190: FutureWarning: The join_axes-keyword is deprecated. Use .reindex or .reindex_like on the result to achieve the same functionality.
08-14-2021 17:02:53.975 ERROR ChunkedExternProcessor [12592 ChunkedExternProcessorStderrLogger] - stderr: merged_df = pd.concat([original_df, additional_df], axis=1, join_axes=[original_df.index])
08-14-2021 17:02:54.308 ERROR ChunkedExternProcessor [16924 phase_1] - Error in 'fit' command: Error while saving model "example_disk_utilization": Model does not exist
08-14-2021 17:02:54.314 INFO ReducePhaseExecutor [16924 phase_1] - Not downloading remote search.log files. Reason: No remote event providers.
08-14-2021 17:02:54.314 INFO ReducePhaseExecutor [16924 phase_1] - Not downloading remote search_telemetry.json files. Reason: No remote event providers.
08-14-2021 17:02:54.314 INFO ReducePhaseExecutor [16924 phase_1] - Ending phase_1
08-14-2021 17:02:54.314 INFO UserManager [16924 phase_1] - Unwound user context: NULL -> NULL
08-14-2021 17:02:54.314 ERROR SearchOrchestrator [7440 searchOrchestrator] - Phase_1 failed due to : Error in 'fit' command: Error while saving model "example_disk_utilization": Model does not exist
08-14-2021 17:02:54.314 INFO ReducePhaseExecutor [8616 StatusEnforcerThread] - ReducePhaseExecutor=1 action=CANCEL
08-14-2021 17:02:54.314 INFO DispatchExecutor [8616 StatusEnforcerThread] - User applied action=CANCEL while status=0
08-14-2021 17:02:54.314 ERROR SearchStatusEnforcer [8616 StatusEnforcerThread] - sid:1628931764.26 Error in 'fit' command: Error while saving model "example_disk_utilization": Model does not exist
08-14-2021 17:02:54.314 INFO SearchStatusEnforcer [8616 StatusEnforcerThread] - State changed to FAILED due to: Error in 'fit' command: Error while saving model "example_disk_utilization": Model does not exist
08-14-2021 17:02:54.332 ERROR ChunkedExternProcessor [12592 ChunkedExternProcessorStderrLogger] - stderr: Traceback (most recent call last):
08-14-2021 17:02:54.332 ERROR ChunkedExternProcessor [12592 ChunkedExternProcessorStderrLogger] - stderr: File "C:\Program Files\Splunk\etc\apps\Splunk_ML_Toolkit\bin\util\lookups_util.py", line 73, in parse_model_reply
08-14-2021 17:02:54.332 ERROR ChunkedExternProcessor [12592 ChunkedExternProcessorStderrLogger] - stderr: return lookups_parse_reply(reply)
08-14-2021 17:02:54.332 ERROR ChunkedExternProcessor [12592 ChunkedExternProcessorStderrLogger] - stderr: File "C:\Program Files\Splunk\etc\apps\Splunk_ML_Toolkit\bin\util\lookups_util.py", line 57, in lookups_parse_reply
08-14-2021 17:02:54.332 ERROR ChunkedExternProcessor [12592 ChunkedExternProcessorStderrLogger] - stderr: raise e
08-14-2021 17:02:54.332 ERROR ChunkedExternProcessor [12592 ChunkedExternProcessorStderrLogger] - stderr: File "C:\Program Files\Splunk\etc\apps\Splunk_ML_Toolkit\bin\util\lookups_util.py", line 47, in lookups_parse_reply
08-14-2021 17:02:54.332 ERROR ChunkedExternProcessor [12592 ChunkedExternProcessorStderrLogger] - stderr: raise LookupNotFoundException()
08-14-2021 17:02:54.332 ERROR ChunkedExternProcessor [12592 ChunkedExternProcessorStderrLogger] - stderr: util.lookup_exceptions.LookupNotFoundException: Lookup does not exist
08-14-2021 17:02:54.332 ERROR ChunkedExternProcessor [12592 ChunkedExternProcessorStderrLogger] - stderr: During handling of the above exception, another exception occurred:
08-14-2021 17:02:54.332 ERROR ChunkedExternProcessor [12592 ChunkedExternProcessorStderrLogger] - stderr: Traceback (most recent call last):
08-14-2021 17:02:54.332 ERROR ChunkedExternProcessor [12592 ChunkedExternProcessorStderrLogger] - stderr: File "C:\Program Files\Splunk\etc\apps\Splunk_ML_Toolkit\bin\processors\FitBatchProcessor.py", line 202, in save_model
08-14-2021 17:02:54.332 ERROR ChunkedExternProcessor [12592 ChunkedExternProcessorStderrLogger] - stderr: namespace=self.namespace,
08-14-2021 17:02:54.332 ERROR ChunkedExternProcessor [12592 ChunkedExternProcessorStderrLogger] - stderr: File "C:\Program Files\Splunk\etc\apps\Splunk_ML_Toolkit\bin\models\base.py", line 177, in save_model
08-14-2021 17:02:54.332 ERROR ChunkedExternProcessor [12592 ChunkedExternProcessorStderrLogger] - stderr: parse_model_reply(reply)
08-14-2021 17:02:54.332 ERROR ChunkedExternProcessor [12592 ChunkedExternProcessorStderrLogger] - stderr: File "C:\Program Files\Splunk\etc\apps\Splunk_ML_Toolkit\bin\util\lookups_util.py", line 75, in parse_model_reply
08-14-2021 17:02:54.332 ERROR ChunkedExternProcessor [12592 ChunkedExternProcessorStderrLogger] - stderr: raise ModelNotFoundException()
08-14-2021 17:02:54.332 ERROR ChunkedExternProcessor [12592 ChunkedExternProcessorStderrLogger] - stderr: util.lookup_exceptions.ModelNotFoundException: Model does not exist
08-14-2021 17:02:54.332 ERROR ChunkedExternProcessor [12592 ChunkedExternProcessorStderrLogger] - stderr: Error while saving model "example_disk_utilization": Model does not exist
08-14-2021 17:02:54.332 ERROR ChunkedExternProcessor [12592 ChunkedExternProcessorStderrLogger] - stderr: Traceback (most recent call last):
08-14-2021 17:02:54.332 ERROR ChunkedExternProcessor [12592 ChunkedExternProcessorStderrLogger] - stderr: File "C:\Program Files\Splunk\etc\apps\Splunk_ML_Toolkit\bin\util\lookups_util.py", line 73, in parse_model_reply
08-14-2021 17:02:54.332 ERROR ChunkedExternProcessor [12592 ChunkedExternProcessorStderrLogger] - stderr: return lookups_parse_reply(reply)
08-14-2021 17:02:54.332 ERROR ChunkedExternProcessor [12592 ChunkedExternProcessorStderrLogger] - stderr: File "C:\Program Files\Splunk\etc\apps\Splunk_ML_Toolkit\bin\util\lookups_util.py", line 57, in lookups_parse_reply
08-14-2021 17:02:54.332 ERROR ChunkedExternProcessor [12592 ChunkedExternProcessorStderrLogger] - stderr: raise e
08-14-2021 17:02:54.332 ERROR ChunkedExternProcessor [12592 ChunkedExternProcessorStderrLogger] - stderr: File "C:\Program Files\Splunk\etc\apps\Splunk_ML_Toolkit\bin\util\lookups_util.py", line 47, in lookups_parse_reply
08-14-2021 17:02:54.332 ERROR ChunkedExternProcessor [12592 ChunkedExternProcessorStderrLogger] - stderr: raise LookupNotFoundException()
08-14-2021 17:02:54.332 ERROR ChunkedExternProcessor [12592 ChunkedExternProcessorStderrLogger] - stderr: util.lookup_exceptions.LookupNotFoundException: Lookup does not exist
08-14-2021 17:02:54.332 ERROR ChunkedExternProcessor [12592 ChunkedExternProcessorStderrLogger] - stderr: During handling of the above exception, another exception occurred:
08-14-2021 17:02:54.332 ERROR ChunkedExternProcessor [12592 ChunkedExternProcessorStderrLogger] - stderr: Traceback (most recent call last):
08-14-2021 17:02:54.332 ERROR ChunkedExternProcessor [12592 ChunkedExternProcessorStderrLogger] - stderr: File "C:\Program Files\Splunk\etc\apps\Splunk_ML_Toolkit\bin\processors\FitBatchProcessor.py", line 202, in save_model
08-14-2021 17:02:54.332 ERROR ChunkedExternProcessor [12592 ChunkedExternProcessorStderrLogger] - stderr: namespace=self.namespace,
08-14-2021 17:02:54.332 ERROR ChunkedExternProcessor [12592 ChunkedExternProcessorStderrLogger] - stderr: File "C:\Program Files\Splunk\etc\apps\Splunk_ML_Toolkit\bin\models\base.py", line 177, in save_model
08-14-2021 17:02:54.332 ERROR ChunkedExternProcessor [12592 ChunkedExternProcessorStderrLogger] - stderr: parse_model_reply(reply)
08-14-2021 17:02:54.332 ERROR ChunkedExternProcessor [12592 ChunkedExternProcessorStderrLogger] - stderr: File "C:\Program Files\Splunk\etc\apps\Splunk_ML_Toolkit\bin\util\lookups_util.py", line 75, in parse_model_reply
08-14-2021 17:02:54.332 ERROR ChunkedExternProcessor [12592 ChunkedExternProcessorStderrLogger] - stderr: raise ModelNotFoundException()
08-14-2021 17:02:54.332 ERROR ChunkedExternProcessor [12592 ChunkedExternProcessorStderrLogger] - stderr: util.lookup_exceptions.ModelNotFoundException: Model does not exist
08-14-2021 17:02:54.332 ERROR ChunkedExternProcessor [12592 ChunkedExternProcessorStderrLogger] - stderr: During handling of the above exception, another exception occurred:
08-14-2021 17:02:54.332 ERROR ChunkedExternProcessor [12592 ChunkedExternProcessorStderrLogger] - stderr: Traceback (most recent call last):
08-14-2021 17:02:54.332 ERROR ChunkedExternProcessor [12592 ChunkedExternProcessorStderrLogger] - stderr: File "C:\Program Files\Splunk\etc\apps\Splunk_ML_Toolkit\bin\cexc\__init__.py", line 174, in run
08-14-2021 17:02:54.332 ERROR ChunkedExternProcessor [12592 ChunkedExternProcessorStderrLogger] - stderr: while self._handle_chunk():
08-14-2021 17:02:54.332 ERROR ChunkedExternProcessor [12592 ChunkedExternProcessorStderrLogger] - stderr: File "C:\Program Files\Splunk\etc\apps\Splunk_ML_Toolkit\bin\cexc\__init__.py", line 236, in _handle_chunk
08-14-2021 17:02:54.332 ERROR ChunkedExternProcessor [12592 ChunkedExternProcessorStderrLogger] - stderr: ret = self.handler(metadata, body)
08-14-2021 17:02:54.332 ERROR ChunkedExternProcessor [12592 ChunkedExternProcessorStderrLogger] - stderr: File "C:\Program Files\Splunk\etc\apps\Splunk_ML_Toolkit\bin\fit.py", line 154, in handler
08-14-2021 17:02:54.332 ERROR ChunkedExternProcessor [12592 ChunkedExternProcessorStderrLogger] - stderr: self.controller.finalize()
08-14-2021 17:02:54.332 ERROR ChunkedExternProcessor [12592 ChunkedExternProcessorStderrLogger] - stderr: File "C:\Program Files\Splunk\etc\apps\Splunk_ML_Toolkit\bin\chunked_controller.py", line 241, in finalize
08-14-2021 17:02:54.332 ERROR ChunkedExternProcessor [12592 ChunkedExternProcessorStderrLogger] - stderr: self.processor.save_model()
08-14-2021 17:02:54.332 ERROR ChunkedExternProcessor [12592 ChunkedExternProcessorStderrLogger] - stderr: File "C:\Program Files\Splunk\etc\apps\Splunk_ML_Toolkit\bin\processors\FitBatchProcessor.py", line 207, in save_model
08-14-2021 17:02:54.332 ERROR ChunkedExternProcessor [12592 ChunkedExternProcessorStderrLogger] - stderr: 'Error while saving model "%s": %s' % (self.algo_options['model_name'], e)
08-14-2021 17:02:54.333 ERROR ChunkedExternProcessor [12592 ChunkedExternProcessorStderrLogger] - stderr: RuntimeError: Error while saving model "example_disk_utilization": Model does not exist
08-14-2021 17:02:54.343 INFO UserManager [8616 StatusEnforcerThread] - Unwound user context: NULL -> NULL
08-14-2021 17:02:54.345 INFO DispatchStorageManager [7440 searchOrchestrator] - Remote storage disabled for search artifacts.
08-14-2021 17:02:54.345 INFO DispatchManager [7440 searchOrchestrator] - DispatchManager::dispatchHasFinished(id='1628931764.26', username='admin')
08-14-2021 17:02:54.346 INFO UserManager [7440 searchOrchestrator] - Unwound user context: NULL -> NULL
08-14-2021 17:02:54.410 INFO SearchStatusEnforcer [13600 RunDispatch] - SearchStatusEnforcer is already terminated
08-14-2021 17:02:54.410 INFO UserManager [13600 RunDispatch] - Unwound user context: NULL -> NULL
08-14-2021 17:02:54.410 INFO LookupDataProvider [13600 RunDispatch] - Clearing out lookup shared provider map
08-14-2021 17:02:54.436 INFO ShutdownHandler [15220 Shutdown] - Shutting down splunkd
----
08-14-2021 17:02:54.436 INFO ShutdownHandler [15220 Shutdown] - Shutdown complete in 0 microseconds
08-14-2021 17:02:54.438 INFO HealthReporter [15936 MainThread] - aggregate_ingestion_latency_health with value=0 from stanza=health_reporter will disable the aggregation of ingestion latency health reporter.
08-14-2021 17:02:54.438 ERROR dispatchRunner [15936 MainThread] - RunDispatch::runDispatchThread threw error: Error in 'fit' command: Error while saving model "example_disk_utilization": Model does not exist