All Apps and Add-ons

MLTK breaks when using DEVTEST license (Error while saving model.)

davidjaniec
Explorer

I'm running Splunk 8.2.1 in DEVTEST to learn how to use MLTK. Trial and Free licenses work, DEVTEST does not work.

I left the license in trial mode, at this point I confirmed that MLTK was able to generate results for a few of the showcase items (disk space utilisation etc). Once I installed my DEVTEST license, the error is reported:

Error in 'fit' command: Error while saving model "example_disk_utilization": Model does not exist.

I switched the Splunk instance to Free mode, ran the example (Predict Disk Utilization) again and it began working again.

Below the search.log is included. Any suggestions/advice is welcomed.

Dave

08-14-2021 17:02:44.511 INFO dispatchRunner [0 MainThread] - Search process mode: freestanding (build ddff1c41e5cf).
08-14-2021 17:02:44.511 INFO dispatchRunner [0 MainThread] - initing LicenseMgr in search process: nonPro=1
08-14-2021 17:02:44.511 INFO LMStackMgr [0 MainThread] - Initializing CleMgr...
08-14-2021 17:02:44.512 INFO LicenseMgr [0 MainThread] - Initing LicenseMgr
08-14-2021 17:02:44.515 INFO ServerConfig [0 MainThread] - Found no hostname options in server.conf. Will attempt to use default for now.
08-14-2021 17:02:44.515 INFO ServerConfig [0 MainThread] - Host name option is "".
08-14-2021 17:02:44.530 INFO ServerConfig [0 MainThread] - SSL session cache path enabled 0 session timeout on SSL server 300.000
08-14-2021 17:02:44.532 INFO ServerConfig [0 MainThread] - Splunk is starting with EC-SSC disabled
08-14-2021 17:02:44.532 INFO LMConfig [0 MainThread] - serverName=homeserver guid=xxxxxxxxxxxxxxxxxxxxxxxx
08-14-2021 17:02:44.532 INFO LMConfig [0 MainThread] - connection_timeout=30
08-14-2021 17:02:44.532 INFO LMConfig [0 MainThread] - send_timeout=30
08-14-2021 17:02:44.532 INFO LMConfig [0 MainThread] - receive_timeout=30
08-14-2021 17:02:44.532 INFO LMConfig [0 MainThread] - key=license_warnings_update_interval not found in licenser stanza of server.conf, defaulting=0
08-14-2021 17:02:44.532 INFO LMConfig [0 MainThread] - squash_threshold=2000
08-14-2021 17:02:44.532 INFO LMConfig [0 MainThread] - strict_pool_quota=1
08-14-2021 17:02:44.532 INFO LMConfig [0 MainThread] - key=pool_suggestion not found in licenser stanza of server.conf, defaulting=''
08-14-2021 17:02:44.532 INFO LMConfig [0 MainThread] - key=test_aws_metering not found in licenser stanza of server.conf, defaulting=0
08-14-2021 17:02:44.532 INFO LMConfig [0 MainThread] - key=test_aws_product_code not found in licenser stanza of server.conf, defaulting=0
08-14-2021 17:02:44.532 INFO LicenseMgr [0 MainThread] - Initing LicenseMgr runContext_splunkd=false
08-14-2021 17:02:44.532 INFO LMStackMgr [0 MainThread] - closing stack mgr
08-14-2021 17:02:44.532 INFO LMSlaveInfo [0 MainThread] - all slaves cleared
08-14-2021 17:02:44.532 INFO LMStackMgr [0 MainThread] - Initalized license_warnings_update_interval=auto
08-14-2021 17:02:44.532 INFO LMStackMgr [0 MainThread] - License Manager supports Conditional Licensing Enforcement. For baked in CLE policies, window_period=60 days, max_violations=45, for stack size below 107374182400 bytes
08-14-2021 17:02:44.532 INFO LMLicense [0 MainThread] - Applying default enforcement policy for free
08-14-2021 17:02:44.532 INFO LMStackMgr [0 MainThread] - Added policy WinSz=30 Warnings=3 MaxSize=0 isDefault=1 features= for free
08-14-2021 17:02:44.532 INFO LMLicense [0 MainThread] - Applying default enforcement policy for forwarder
08-14-2021 17:02:44.532 INFO LMStackMgr [0 MainThread] - Added policy WinSz=30 Warnings=5 MaxSize=0 isDefault=1 features= for forwarder
08-14-2021 17:02:44.532 INFO LMStackMgr [0 MainThread] - Skipping trial license as alternative license type in use
08-14-2021 17:02:44.534 INFO LMStack [0 MainThread] - Added type=enterprise license, from file=Splunk (2).License.lic, to stack=enterprise of group=Enterprise
08-14-2021 17:02:44.534 INFO LMLicense [0 MainThread] - Applying default CLE policy for enterprise
08-14-2021 17:02:44.534 INFO LMStackMgr [0 MainThread] - created stack='enterprise'
08-14-2021 17:02:44.534 INFO LMStackMgr [0 MainThread] - Added policy WinSz=60 Warnings=45 MaxSize=107374182400 isDefault=1 features=LocalSearch for enterprise
08-14-2021 17:02:44.534 INFO LMStackMgr [0 MainThread] - Skipping trial pool stanza as alternative license in use
08-14-2021 17:02:44.534 INFO LMStackMgr [0 MainThread] - added pool auto_generated_pool_enterprise to stack enterprise
08-14-2021 17:02:44.534 INFO LMStackMgr [0 MainThread] - added pool auto_generated_pool_forwarder to stack forwarder
08-14-2021 17:02:44.534 INFO LMStackMgr [0 MainThread] - added pool auto_generated_pool_free to stack free
08-14-2021 17:02:44.534 INFO LMStackMgr [0 MainThread] - Initialized hideQuotaWarning = "0"
08-14-2021 17:02:44.534 INFO LMStackMgr [0 MainThread] - init completed [xxxxxxxxxxxxxxxxxxxxxx,Enterprise,runContext_splunkd=false]
08-14-2021 17:02:44.534 INFO LicenseMgr [0 MainThread] - StackMgr init complete...
08-14-2021 17:02:44.534 INFO LMTracker [0 MainThread] - Setting default product type='enterprise'
08-14-2021 17:02:44.534 INFO LMTracker [0 MainThread] - this is not splunkd, will perform partial init
08-14-2021 17:02:44.534 INFO LicenseMgr [0 MainThread] - Tracker init complete...

----

08-14-2021 17:02:52.634 INFO DispatchExecutor [16924 phase_1] - BEGIN OPEN: Processor=inputlookup
08-14-2021 17:02:52.634 INFO DispatchExecutor [16924 phase_1] - END OPEN: Processor=inputlookup
08-14-2021 17:02:52.634 INFO DispatchExecutor [16924 phase_1] - BEGIN OPEN: Processor=fit
08-14-2021 17:02:52.688 INFO SearchOperator:inputcsv [16924 phase_1] - sid:1628931764.26 Successfully read lookup file 'C:\Program Files\Splunk\etc\apps\Splunk_ML_Toolkit\lookups\server_power.csv'.
08-14-2021 17:02:52.688 INFO PreviewExecutor [8616 StatusEnforcerThread] - Preview Enforcing initialization done
08-14-2021 17:02:52.689 INFO ReducePhaseExecutor [8616 StatusEnforcerThread] - ReducePhaseExecutor=1 action=PREVIEW
08-14-2021 17:02:52.690 INFO DispatchExecutor [16924 phase_1] - END OPEN: Processor=fit
08-14-2021 17:02:53.975 ERROR ChunkedExternProcessor [12592 ChunkedExternProcessorStderrLogger] - stderr: C:\Program Files\Splunk\etc\apps\Splunk_SA_Scientific_Python_windows_x86_64\bin\windows_x86_64\lib\site-packages\sklearn\ensemble\forest.py:245: FutureWarning: The default value of n_estimators will change from 10 in version 0.20 to 100 in 0.22.
08-14-2021 17:02:53.975 ERROR ChunkedExternProcessor [12592 ChunkedExternProcessorStderrLogger] - stderr: "10 in version 0.20 to 100 in 0.22.", FutureWarning)
08-14-2021 17:02:53.975 ERROR ChunkedExternProcessor [12592 ChunkedExternProcessorStderrLogger] - stderr: C:\Program Files\Splunk\etc\apps\Splunk_ML_Toolkit\bin\util\df_util.py:190: FutureWarning: The join_axes-keyword is deprecated. Use .reindex or .reindex_like on the result to achieve the same functionality.
08-14-2021 17:02:53.975 ERROR ChunkedExternProcessor [12592 ChunkedExternProcessorStderrLogger] - stderr: merged_df = pd.concat([original_df, additional_df], axis=1, join_axes=[original_df.index])
08-14-2021 17:02:54.308 ERROR ChunkedExternProcessor [16924 phase_1] - Error in 'fit' command: Error while saving model "example_disk_utilization": Model does not exist
08-14-2021 17:02:54.314 INFO ReducePhaseExecutor [16924 phase_1] - Not downloading remote search.log files. Reason: No remote event providers.
08-14-2021 17:02:54.314 INFO ReducePhaseExecutor [16924 phase_1] - Not downloading remote search_telemetry.json files. Reason: No remote event providers.
08-14-2021 17:02:54.314 INFO ReducePhaseExecutor [16924 phase_1] - Ending phase_1
08-14-2021 17:02:54.314 INFO UserManager [16924 phase_1] - Unwound user context: NULL -> NULL
08-14-2021 17:02:54.314 ERROR SearchOrchestrator [7440 searchOrchestrator] - Phase_1 failed due to : Error in 'fit' command: Error while saving model "example_disk_utilization": Model does not exist
08-14-2021 17:02:54.314 INFO ReducePhaseExecutor [8616 StatusEnforcerThread] - ReducePhaseExecutor=1 action=CANCEL
08-14-2021 17:02:54.314 INFO DispatchExecutor [8616 StatusEnforcerThread] - User applied action=CANCEL while status=0
08-14-2021 17:02:54.314 ERROR SearchStatusEnforcer [8616 StatusEnforcerThread] - sid:1628931764.26 Error in 'fit' command: Error while saving model "example_disk_utilization": Model does not exist
08-14-2021 17:02:54.314 INFO SearchStatusEnforcer [8616 StatusEnforcerThread] - State changed to FAILED due to: Error in 'fit' command: Error while saving model "example_disk_utilization": Model does not exist
08-14-2021 17:02:54.332 ERROR ChunkedExternProcessor [12592 ChunkedExternProcessorStderrLogger] - stderr: Traceback (most recent call last):
08-14-2021 17:02:54.332 ERROR ChunkedExternProcessor [12592 ChunkedExternProcessorStderrLogger] - stderr: File "C:\Program Files\Splunk\etc\apps\Splunk_ML_Toolkit\bin\util\lookups_util.py", line 73, in parse_model_reply
08-14-2021 17:02:54.332 ERROR ChunkedExternProcessor [12592 ChunkedExternProcessorStderrLogger] - stderr: return lookups_parse_reply(reply)
08-14-2021 17:02:54.332 ERROR ChunkedExternProcessor [12592 ChunkedExternProcessorStderrLogger] - stderr: File "C:\Program Files\Splunk\etc\apps\Splunk_ML_Toolkit\bin\util\lookups_util.py", line 57, in lookups_parse_reply
08-14-2021 17:02:54.332 ERROR ChunkedExternProcessor [12592 ChunkedExternProcessorStderrLogger] - stderr: raise e
08-14-2021 17:02:54.332 ERROR ChunkedExternProcessor [12592 ChunkedExternProcessorStderrLogger] - stderr: File "C:\Program Files\Splunk\etc\apps\Splunk_ML_Toolkit\bin\util\lookups_util.py", line 47, in lookups_parse_reply
08-14-2021 17:02:54.332 ERROR ChunkedExternProcessor [12592 ChunkedExternProcessorStderrLogger] - stderr: raise LookupNotFoundException()
08-14-2021 17:02:54.332 ERROR ChunkedExternProcessor [12592 ChunkedExternProcessorStderrLogger] - stderr: util.lookup_exceptions.LookupNotFoundException: Lookup does not exist
08-14-2021 17:02:54.332 ERROR ChunkedExternProcessor [12592 ChunkedExternProcessorStderrLogger] - stderr: During handling of the above exception, another exception occurred:
08-14-2021 17:02:54.332 ERROR ChunkedExternProcessor [12592 ChunkedExternProcessorStderrLogger] - stderr: Traceback (most recent call last):
08-14-2021 17:02:54.332 ERROR ChunkedExternProcessor [12592 ChunkedExternProcessorStderrLogger] - stderr: File "C:\Program Files\Splunk\etc\apps\Splunk_ML_Toolkit\bin\processors\FitBatchProcessor.py", line 202, in save_model
08-14-2021 17:02:54.332 ERROR ChunkedExternProcessor [12592 ChunkedExternProcessorStderrLogger] - stderr: namespace=self.namespace,
08-14-2021 17:02:54.332 ERROR ChunkedExternProcessor [12592 ChunkedExternProcessorStderrLogger] - stderr: File "C:\Program Files\Splunk\etc\apps\Splunk_ML_Toolkit\bin\models\base.py", line 177, in save_model
08-14-2021 17:02:54.332 ERROR ChunkedExternProcessor [12592 ChunkedExternProcessorStderrLogger] - stderr: parse_model_reply(reply)
08-14-2021 17:02:54.332 ERROR ChunkedExternProcessor [12592 ChunkedExternProcessorStderrLogger] - stderr: File "C:\Program Files\Splunk\etc\apps\Splunk_ML_Toolkit\bin\util\lookups_util.py", line 75, in parse_model_reply
08-14-2021 17:02:54.332 ERROR ChunkedExternProcessor [12592 ChunkedExternProcessorStderrLogger] - stderr: raise ModelNotFoundException()
08-14-2021 17:02:54.332 ERROR ChunkedExternProcessor [12592 ChunkedExternProcessorStderrLogger] - stderr: util.lookup_exceptions.ModelNotFoundException: Model does not exist
08-14-2021 17:02:54.332 ERROR ChunkedExternProcessor [12592 ChunkedExternProcessorStderrLogger] - stderr: Error while saving model "example_disk_utilization": Model does not exist
08-14-2021 17:02:54.332 ERROR ChunkedExternProcessor [12592 ChunkedExternProcessorStderrLogger] - stderr: Traceback (most recent call last):
08-14-2021 17:02:54.332 ERROR ChunkedExternProcessor [12592 ChunkedExternProcessorStderrLogger] - stderr: File "C:\Program Files\Splunk\etc\apps\Splunk_ML_Toolkit\bin\util\lookups_util.py", line 73, in parse_model_reply
08-14-2021 17:02:54.332 ERROR ChunkedExternProcessor [12592 ChunkedExternProcessorStderrLogger] - stderr: return lookups_parse_reply(reply)
08-14-2021 17:02:54.332 ERROR ChunkedExternProcessor [12592 ChunkedExternProcessorStderrLogger] - stderr: File "C:\Program Files\Splunk\etc\apps\Splunk_ML_Toolkit\bin\util\lookups_util.py", line 57, in lookups_parse_reply
08-14-2021 17:02:54.332 ERROR ChunkedExternProcessor [12592 ChunkedExternProcessorStderrLogger] - stderr: raise e
08-14-2021 17:02:54.332 ERROR ChunkedExternProcessor [12592 ChunkedExternProcessorStderrLogger] - stderr: File "C:\Program Files\Splunk\etc\apps\Splunk_ML_Toolkit\bin\util\lookups_util.py", line 47, in lookups_parse_reply
08-14-2021 17:02:54.332 ERROR ChunkedExternProcessor [12592 ChunkedExternProcessorStderrLogger] - stderr: raise LookupNotFoundException()
08-14-2021 17:02:54.332 ERROR ChunkedExternProcessor [12592 ChunkedExternProcessorStderrLogger] - stderr: util.lookup_exceptions.LookupNotFoundException: Lookup does not exist
08-14-2021 17:02:54.332 ERROR ChunkedExternProcessor [12592 ChunkedExternProcessorStderrLogger] - stderr: During handling of the above exception, another exception occurred:
08-14-2021 17:02:54.332 ERROR ChunkedExternProcessor [12592 ChunkedExternProcessorStderrLogger] - stderr: Traceback (most recent call last):
08-14-2021 17:02:54.332 ERROR ChunkedExternProcessor [12592 ChunkedExternProcessorStderrLogger] - stderr: File "C:\Program Files\Splunk\etc\apps\Splunk_ML_Toolkit\bin\processors\FitBatchProcessor.py", line 202, in save_model
08-14-2021 17:02:54.332 ERROR ChunkedExternProcessor [12592 ChunkedExternProcessorStderrLogger] - stderr: namespace=self.namespace,
08-14-2021 17:02:54.332 ERROR ChunkedExternProcessor [12592 ChunkedExternProcessorStderrLogger] - stderr: File "C:\Program Files\Splunk\etc\apps\Splunk_ML_Toolkit\bin\models\base.py", line 177, in save_model
08-14-2021 17:02:54.332 ERROR ChunkedExternProcessor [12592 ChunkedExternProcessorStderrLogger] - stderr: parse_model_reply(reply)
08-14-2021 17:02:54.332 ERROR ChunkedExternProcessor [12592 ChunkedExternProcessorStderrLogger] - stderr: File "C:\Program Files\Splunk\etc\apps\Splunk_ML_Toolkit\bin\util\lookups_util.py", line 75, in parse_model_reply
08-14-2021 17:02:54.332 ERROR ChunkedExternProcessor [12592 ChunkedExternProcessorStderrLogger] - stderr: raise ModelNotFoundException()
08-14-2021 17:02:54.332 ERROR ChunkedExternProcessor [12592 ChunkedExternProcessorStderrLogger] - stderr: util.lookup_exceptions.ModelNotFoundException: Model does not exist
08-14-2021 17:02:54.332 ERROR ChunkedExternProcessor [12592 ChunkedExternProcessorStderrLogger] - stderr: During handling of the above exception, another exception occurred:
08-14-2021 17:02:54.332 ERROR ChunkedExternProcessor [12592 ChunkedExternProcessorStderrLogger] - stderr: Traceback (most recent call last):
08-14-2021 17:02:54.332 ERROR ChunkedExternProcessor [12592 ChunkedExternProcessorStderrLogger] - stderr: File "C:\Program Files\Splunk\etc\apps\Splunk_ML_Toolkit\bin\cexc\__init__.py", line 174, in run
08-14-2021 17:02:54.332 ERROR ChunkedExternProcessor [12592 ChunkedExternProcessorStderrLogger] - stderr: while self._handle_chunk():
08-14-2021 17:02:54.332 ERROR ChunkedExternProcessor [12592 ChunkedExternProcessorStderrLogger] - stderr: File "C:\Program Files\Splunk\etc\apps\Splunk_ML_Toolkit\bin\cexc\__init__.py", line 236, in _handle_chunk
08-14-2021 17:02:54.332 ERROR ChunkedExternProcessor [12592 ChunkedExternProcessorStderrLogger] - stderr: ret = self.handler(metadata, body)
08-14-2021 17:02:54.332 ERROR ChunkedExternProcessor [12592 ChunkedExternProcessorStderrLogger] - stderr: File "C:\Program Files\Splunk\etc\apps\Splunk_ML_Toolkit\bin\fit.py", line 154, in handler
08-14-2021 17:02:54.332 ERROR ChunkedExternProcessor [12592 ChunkedExternProcessorStderrLogger] - stderr: self.controller.finalize()
08-14-2021 17:02:54.332 ERROR ChunkedExternProcessor [12592 ChunkedExternProcessorStderrLogger] - stderr: File "C:\Program Files\Splunk\etc\apps\Splunk_ML_Toolkit\bin\chunked_controller.py", line 241, in finalize
08-14-2021 17:02:54.332 ERROR ChunkedExternProcessor [12592 ChunkedExternProcessorStderrLogger] - stderr: self.processor.save_model()
08-14-2021 17:02:54.332 ERROR ChunkedExternProcessor [12592 ChunkedExternProcessorStderrLogger] - stderr: File "C:\Program Files\Splunk\etc\apps\Splunk_ML_Toolkit\bin\processors\FitBatchProcessor.py", line 207, in save_model
08-14-2021 17:02:54.332 ERROR ChunkedExternProcessor [12592 ChunkedExternProcessorStderrLogger] - stderr: 'Error while saving model "%s": %s' % (self.algo_options['model_name'], e)
08-14-2021 17:02:54.333 ERROR ChunkedExternProcessor [12592 ChunkedExternProcessorStderrLogger] - stderr: RuntimeError: Error while saving model "example_disk_utilization": Model does not exist
08-14-2021 17:02:54.343 INFO UserManager [8616 StatusEnforcerThread] - Unwound user context: NULL -> NULL
08-14-2021 17:02:54.345 INFO DispatchStorageManager [7440 searchOrchestrator] - Remote storage disabled for search artifacts.
08-14-2021 17:02:54.345 INFO DispatchManager [7440 searchOrchestrator] - DispatchManager::dispatchHasFinished(id='1628931764.26', username='admin')
08-14-2021 17:02:54.346 INFO UserManager [7440 searchOrchestrator] - Unwound user context: NULL -> NULL
08-14-2021 17:02:54.410 INFO SearchStatusEnforcer [13600 RunDispatch] - SearchStatusEnforcer is already terminated
08-14-2021 17:02:54.410 INFO UserManager [13600 RunDispatch] - Unwound user context: NULL -> NULL
08-14-2021 17:02:54.410 INFO LookupDataProvider [13600 RunDispatch] - Clearing out lookup shared provider map
08-14-2021 17:02:54.436 INFO ShutdownHandler [15220 Shutdown] - Shutting down splunkd

----


08-14-2021 17:02:54.436 INFO ShutdownHandler [15220 Shutdown] - Shutdown complete in 0 microseconds
08-14-2021 17:02:54.438 INFO HealthReporter [15936 MainThread] - aggregate_ingestion_latency_health with value=0 from stanza=health_reporter will disable the aggregation of ingestion latency health reporter.
08-14-2021 17:02:54.438 ERROR dispatchRunner [15936 MainThread] - RunDispatch::runDispatchThread threw error: Error in 'fit' command: Error while saving model "example_disk_utilization": Model does not exist

 

Labels (2)
0 Karma
Get Updates on the Splunk Community!

New Splunk Observability innovations: Deeper visibility and smarter alerting to ...

You asked, we delivered. Splunk Observability Cloud has several new innovations giving you deeper visibility ...

Synthetic Monitoring: Not your Grandma’s Polyester! Tech Talk: DevOps Edition

Register today and join TekStream on Tuesday, February 28 at 11am PT/2pm ET for a demonstration of Splunk ...

Instrumenting Java Websocket Messaging

Instrumenting Java Websocket MessagingThis article is a code-based discussion of passing OpenTelemetry trace ...