Error training model (AI Toolkit) - logexperiment

[Robi2]

Hi, trying out the AI Toolkit App I built some experiments and scheduled training but the training jobs all fail with  "Error in 'logexperiment' command: (JSONDecodeError) Expecting value: line 1 column 1 (char 0) "

The Search looks like this, created from the Experiment
some search | fit somefunction and parameters into "_exp_some-random-id" | logexperiment id=same-random-id

Has the training already occured despite the error? What could be the problem with logexperiment? I wasn't able to find any description of the command.

[Robi2]

Thank you for the pointers, there are some entries in the job log indeed but I don't know which are the important bits:

WARNING <?xml version="1.0" encoding="UTF-8"?>
<response>
<messages>
<msg type="ERROR">KV Store initialization failed. Please contact your system administrator.</msg>
</messages>
</response>
(JSONDecodeError) Expecting value: line 1 column 1 (char 0)

Mongodb is not running  and KV Store complains about expired certificates but I was not able to renew the internal certificate by renamein it and let 'splunk start ' issue a new one but it only complains 

The certificate generation script did not generate the expected certificate file:/opt/splunk/etc/auth/server.pem. Splunkd port communication will not work.
SSL certificate generation failed.

and splunkd-utility.log logs

 ServerConfig - No '/opt/splunk/etc/auth/server.pem' certificate found. Splunkd communication will not work without this. If this is a fresh installation, this should be OK.

Does the logexperiment rely on a functioning kvstore? Is the fit still working and the logexperiment  error could be ignored?

And for the renewing of the certificate (splunk enterprise 9.4 btw) -- am I missing something obvious? Should I better open a support case?

[tscroggins]

Hi @Robi2,

Yes, the logexperiment command uses the experiment_history KV store collection.

The unpublished model will be updated by the fit command before the logexperiment command runs. You can list models with the listmodels command. You should see _exp_<id>. On disk, you should see __mlspl__exp_<id>.mlmodel under $SPLUNK_HOME/etc/users/<user>/Splunk_ML_Toolkit/lookups/ with an updated timestamp. Alerts scheduled within Experiments should automatically reference the retrained, unpublished model using the apply command.

If Splunk fails to automatically regenerate server.pem, make sure the local ca.pem certificate is valid:

/opt/splunk/bin/splunk cmd openssl x509 -in /opt/splunk/etc/auth/ca.pem -dates -noout

It's unlikely the default CA certificate expires before 2027, but the file may have been modified.

If needed, you can manually recreate the certificate with the splunk createssl command:

/opt/splunk/bin/splunk createssl server-cert -d /opt/splunk/etc/auth -n SplunkServerDefaultCert -n $(hostname) -l 2048

Copy /opt/splunk/etc/auth/SplunkServerDefaultCert.pem to /opt/splunk/etc/auth/server.pem.

The SplunkWeb certificate is stored separately from the splunkd certificate.

Note that you shouldn't use default certificates in a production environment. 🙂

[tscroggins]

HI @Robi2,

The exception may have been thrown after the logexperiment command failed to post data to the https://localhost:8089/servicesNS/admin/search/mltk/experiments/<id>/history endpoint.

A stack trace should have been written to the job search.log by the ChunkedExternProcessor logger. Do you see more information in the job log?