All Apps and Add-ons
Highlighted

Splunk Machine Learning Toolkit: error message when importing new algorithm

Engager

I followed the procedure in API Guide 2.4.0 to add the CorrelationMatrix algorithm. But an error resulted.

I am using Windows 10.

I first added [CorrelationMatrix] in algos.conf located at ~\etc\apps\SplunkMLToolkit\default\algos.conf
Then, I added a file CorrelationMatrix.py at ~\etc\apps\SplunkMLToolkit\bin\algos

The .py file was copied from the "Finished example" in the API Guide.

Finally, I restarted Splunk and searched "| inputlookup iris.csv | fit CorrelationMatrix petal* sepal*"

The error message was:

Error in 'fit' command: Error while initializing algorithm "CorrelationMatrix": Algorithm "CorrelationMatrix" cannot be loaded

The system log is:

10-08-2017 11:35:38.579 INFO dispatchRunner - Search process mode: freestanding
10-08-2017 11:35:38.580 INFO dispatchRunner - initing LicenseMgr in search process: nonPro=0
10-08-2017 11:35:38.580 INFO LicenseMgr - Initing LicenseMgr
10-08-2017 11:35:38.583 INFO ServerConfig - Found no hostname options in server.conf. Will attempt to use default for now.
10-08-2017 11:35:38.583 INFO ServerConfig - Host name option is "".
10-08-2017 11:35:38.604 INFO ServerConfig - SSL session cache path enabled 0 session timeout on SSL server 300.000
10-08-2017 11:35:38.606 INFO LMConfig - serverName=xxxxxxxx guid=xxxxxxxxxxxxxxxxxx
10-08-2017 11:35:38.606 INFO LMConfig - connectiontimeout=30
10-08-2017 11:35:38.606 INFO LMConfig - send
timeout=30
10-08-2017 11:35:38.613 INFO LMConfig - receivetimeout=30
10-08-2017 11:35:38.613 INFO LMConfig - squash
threshold=2000
10-08-2017 11:35:38.613 INFO LMConfig - strictpoolquota=1
10-08-2017 11:35:38.613 INFO LMConfig - key=poolsuggestion not found in licenser stanza of server.conf, defaulting=''
10-08-2017 11:35:38.613 INFO LMConfig - key=test
awsmetering not found in licenser stanza of server.conf, defaulting=0
10-08-2017 11:35:38.613 INFO LMConfig - key=test
awsproductcode not found in licenser stanza of server.conf, defaulting=0
10-08-2017 11:35:38.613 INFO LicenseMgr - Initing LicenseMgr runContextsplunkd=false
10-08-2017 11:35:38.613 INFO LMStackMgr - closing stack mgr
10-08-2017 11:35:38.613 INFO LMSlaveInfo - all slaves cleared
10-08-2017 11:35:38.613 INFO LMStackMgr - partial init only since node has remote master=https://splunk.xxxxxx:8089
10-08-2017 11:35:38.613 INFO LicenseMgr - StackMgr init complete...
10-08-2017 11:35:38.613 INFO LMTracker - Setting default product type='enterprise'
10-08-2017 11:35:38.613 INFO LMTracker - this is not splunkd, will perform partial init
10-08-2017 11:35:38.613 INFO LMTracker - Setting feature=Acceleration state=ENABLED (featureStatus=1)
10-08-2017 11:35:38.613 INFO LMTracker - Setting feature=AdvancedSearchCommands state=ENABLED (featureStatus=1)
10-08-2017 11:35:38.613 INFO LMTracker - Setting feature=AdvancedXML state=ENABLED (featureStatus=1)
10-08-2017 11:35:38.613 INFO LMTracker - Setting feature=Alerting state=ENABLED (featureStatus=1)
10-08-2017 11:35:38.613 INFO LMTracker - Setting feature=ArchiveToHdfs state=ENABLED (featureStatus=1)
10-08-2017 11:35:38.613 INFO LMTracker - Setting feature=Auth state=ENABLED (featureStatus=1)
10-08-2017 11:35:38.613 INFO LMTracker - Setting feature=CustomRoles state=ENABLED (featureStatus=1)
10-08-2017 11:35:38.613 INFO LMTracker - Setting feature=DeployClient state=ENABLED (featureStatus=1)
10-08-2017 11:35:38.613 INFO LMTracker - Setting feature=DeployServer state=ENABLED (featureStatus=1)
10-08-2017 11:35:38.613 INFO LMTracker - Setting feature=DistSearch state=ENABLED (featureStatus=1)
10-08-2017 11:35:38.613 INFO LMTracker - Setting feature=FwdData state=ENABLED (featureStatus=1)
10-08-2017 11:35:38.613 INFO LMTracker - Setting feature=GuestPass state=ENABLED (featureStatus=1)
10-08-2017 11:35:38.613 INFO LMTracker - Setting feature=KVStore state=ENABLED (featureStatus=1)
10-08-2017 11:35:38.613 INFO LMTracker - Setting feature=LDAPAuth state=ENABLED (featureStatus=1)
10-08-2017 11:35:38.613 INFO LMTracker - Setting feature=LocalSearch state=ENABLED (featureStatus=1)
10-08-2017 11:35:38.613 INFO LMTracker - Setting feature=MultifactorAuth state=ENABLED (featureStatus=1)
10-08-2017 11:35:38.613 INFO LMTracker - Setting feature=MultisiteClustering state=ENABLED (featureStatus=1)
10-08-2017 11:35:38.613 INFO LMTracker - Setting feature=NontableLookups state=ENABLED (featureStatus=1)
10-08-2017 11:35:38.613 INFO LMTracker - Setting feature=RcvData state=ENABLED (featureStatus=1)
10-08-2017 11:35:38.613 INFO LMTracker - Setting feature=RcvSearch state=ENABLED (featureStatus=1)
10-08-2017 11:35:38.613 INFO LMTracker - Setting feature=RollingWindowAlerts state=ENABLED (featureStatus=1)
10-08-2017 11:35:38.613 INFO LMTracker - Setting feature=SAMLAuth state=ENABLED (featureStatus=1)
10-08-2017 11:35:38.613 INFO LMTracker - Setting feature=ScheduledAlerts state=ENABLED (featureStatus=1)
10-08-2017 11:35:38.613 INFO LMTracker - Setting feature=ScheduledReports state=ENABLED (featureStatus=1)
10-08-2017 11:35:38.613 INFO LMTracker - Setting feature=ScheduledSearch state=ENABLED (featureStatus=1)
10-08-2017 11:35:38.613 INFO LMTracker - Setting feature=ScriptedAuth state=ENABLED (featureStatus=1)
10-08-2017 11:35:38.613 INFO LMTracker - Setting feature=SearchheadPooling state=ENABLED (featureStatus=1)
10-08-2017 11:35:38.613 INFO LMTracker - Setting feature=SigningProcessor state=ENABLED (featureStatus=1)
10-08-2017 11:35:38.613 INFO LMTracker - Setting feature=SplunkWeb state=ENABLED (featureStatus=1)
10-08-2017 11:35:38.613 INFO LMTracker - Setting feature=SubgroupId state=ENABLED (featureStatus=1)
10-08-2017 11:35:38.613 INFO LMTracker - Setting feature=SyslogOutputProcessor state=ENABLED (featureStatus=1)
10-08-2017 11:35:38.613 INFO LMTracker - Setting feature=UnisiteClustering state=ENABLED (featureStatus=1)
10-08-2017 11:35:38.613 INFO LicenseMgr - Tracker init complete...
10-08-2017 11:35:38.618 INFO AdminManagerDispatch - added factory for admin handler: 'licenses'
10-08-2017 11:35:38.618 INFO AdminManagerDispatch - added factory for admin handler: 'pools'
10-08-2017 11:35:38.618 INFO AdminManagerDispatch - added factory for admin handler: 'stacks'
10-08-2017 11:35:38.618 INFO AdminManagerDispatch - added factory for admin handler: 'groups'
10-08-2017 11:35:38.618 INFO AdminManagerDispatch - added factory for admin handler: 'slaves'
10-08-2017 11:35:38.618 INFO AdminManagerDispatch - added factory for admin handler: 'localslave'
10-08-2017 11:35:38.618 INFO AdminManagerDispatch - added factory for admin handler: 'licensermessages'
10-08-2017 11:35:38.618 INFO AdminManagerDispatch - added factory for admin handler: 'scriptedwarning'
10-08-2017 11:35:38.618 INFO AdminManagerDispatch - added factory for admin handler: 'licenseusage'
10-08-2017 11:35:38.618 INFO dispatchRunner - Per-process handle limit is 512
10-08-2017 11:35:38.618 INFO dispatchRunner - Increasing per-process handle limit from '512' to '2048'
10-08-2017 11:35:38.618 INFO dispatchRunner - Successfully increased per-process handle limit from '512' to '2048'
10-08-2017 11:35:38.621 INFO dispatchRunner - registering build time modules, count=1
10-08-2017 11:35:38.621 INFO dispatchRunner - registering search time components of build time module name=vix
10-08-2017 11:35:38.621 INFO dispatchRunner - Splunkd starting (build e21ee54bc796).
10-08-2017 11:35:38.621 INFO dispatchRunner - System info: Windows, xxxxxx, 3, 6, x64.
10-08-2017 11:35:38.621 INFO dispatchRunner - Detected 4 (virtual) CPUs, 2 CPU cores, and 16267MB RAM
10-08-2017 11:35:38.621 INFO dispatchRunner - Maximum number of threads (approximate): 8133
10-08-2017 11:35:38.621 INFO dispatchRunner - Arguments are: "search" "--id=xxxxxxxx" "--maxbuckets=0" "--ttl=600" "--maxout=500000" "--maxtime=8640000" "--lookups=1" "--reduce
freq=10" "--user=admin" "--pro" "--roles=admin:candelete:power:splunk-system-role:user"
10-08-2017 11:35:38.621 INFO dispatchRunner - Getting search configuration data from: C:\Program Files\Splunk\etc\modules\parsing\config.xml
10-08-2017 11:35:38.622 INFO BundlesSetup - Setup stats for C:\Program Files\Splunk\etc: wallclock
elapsedmsec=34, cputimeused=0.03125, sharedservicesgeneration=1, sharedservicespopulation=1
10-08-2017 11:35:38.625 INFO UserManagerPro - Load authentication: forcing roles="admin, can
delete, power, splunk-system-role, user"
10-08-2017 11:35:38.626 INFO SessionManager - auth tokens will be generated with shpooling shared secret
10-08-2017 11:35:38.626 INFO UserManager - Setting user context: splunk-system-user
10-08-2017 11:35:38.626 INFO UserManager - Done setting user context: NULL -> splunk-system-user
10-08-2017 11:35:38.626 INFO UserManager - Unwound user context: splunk-system-user -> NULL
10-08-2017 11:35:38.627 INFO UserManager - Setting user context: admin
10-08-2017 11:35:38.627 INFO UserManager - Done setting user context: NULL -> admin
10-08-2017 11:35:38.627 INFO dispatchRunner - search context: user="admin", app="SplunkMLToolkit", bs-pathname="C:\Program Files\Splunk\etc"
10-08-2017 11:35:38.631 INFO dispatchRunner - Executing the DispatchThread.
10-08-2017 11:35:38.631 INFO SearchParser - PARSING: | inputlookup iris.csv | fit CorrelationMatrix petal* sepal*
10-08-2017 11:35:38.640 INFO ISplunkDispatch - Not running in splunkd. Bundle replication not triggered.
10-08-2017 11:35:38.644 INFO UserManager - Setting user context: admin
10-08-2017 11:35:38.644 INFO UserManager - Done setting user context: NULL -> admin
10-08-2017 11:35:38.652 INFO ChunkedExternProcessor - Running process: "C:\Program Files\Splunk\bin\python.exe" "C:\Program Files\Splunk\etc\apps\SplunkMLToolkit\bin\fit.py"
10-08-2017 11:35:38.919 INFO ChunkedExternProcessor - stderr: Running C:\Program Files\Splunk\etc\apps\SplunkSAScientificPythonwindowsx8664\bin\windowsx8664\python.exe C:\Program Files\Splunk\etc\apps\SplunkMLToolkit\bin\fit.py
10-08-2017 11:35:40.554 ERROR ChunkedExternProcessor - Error in 'fit' command: Error while initializing algorithm "CorrelationMatrix": Algorithm "CorrelationMatrix" cannot be loaded
10-08-2017 11:35:40.554 INFO UserManager - Unwound user context: admin -> NULL
10-08-2017 11:35:40.563 INFO UserManager - Setting user context: admin
10-08-2017 11:35:40.563 INFO UserManager - Done setting user context: NULL -> admin
10-08-2017 11:35:40.563 INFO UserManager - Unwound user context: admin -> NULL
10-08-2017 11:35:40.563 INFO DispatchManager - DispatchManager::dispatchHasFinished(id='xxxxxxxxx', username='admin')
10-08-2017 11:35:40.952 INFO UserManager - Unwound user context: admin -> NULL
10-08-2017 11:35:40.958 INFO ShutdownHandler - Shutting down splunkd
10-08-2017 11:35:40.958 INFO ShutdownHandler - shutting down level "ShutdownLevelBegin"
10-08-2017 11:35:40.958 INFO ShutdownHandler - shutting down level "ShutdownLevel
FileIntegrityChecker"
10-08-2017 11:35:40.958 INFO ShutdownHandler - shutting down level "ShutdownLevelJustBeforeKVStore"
10-08-2017 11:35:40.958 INFO ShutdownHandler - shutting down level "ShutdownLevel
KVStore"
10-08-2017 11:35:40.958 INFO ShutdownHandler - shutting down level "ShutdownLevelThruput"
10-08-2017 11:35:40.958 INFO ShutdownHandler - shutting down level "ShutdownLevel
TcpInput1"
10-08-2017 11:35:40.958 INFO ShutdownHandler - shutting down level "ShutdownLevelExecSendInitialSigterm"
10-08-2017 11:35:40.958 INFO ShutdownHandler - shutting down level "ShutdownLevel
TcpOutput"
10-08-2017 11:35:40.958 INFO ShutdownHandler - shutting down level "ShutdownLevelUdpInput"
10-08-2017 11:35:40.958 INFO ShutdownHandler - shutting down level "ShutdownLevel
FifoInput"
10-08-2017 11:35:40.958 INFO ShutdownHandler - shutting down level "ShutdownLevelWinEventLogInput"
10-08-2017 11:35:40.958 INFO ShutdownHandler - shutting down level "ShutdownLevel
HttpInput"
10-08-2017 11:35:40.958 INFO ShutdownHandler - shutting down level "ShutdownLevelCacheManager"
10-08-2017 11:35:40.958 INFO ShutdownHandler - shutting down level "ShutdownLevel
Scheduler"
10-08-2017 11:35:40.958 INFO ShutdownHandler - shutting down level "ShutdownLevelSyslogOutput"
10-08-2017 11:35:40.958 INFO ShutdownHandler - shutting down level "ShutdownLevel
HTTPOutput"
10-08-2017 11:35:40.958 INFO ShutdownHandler - shutting down level "ShutdownLevelArchiveAndOneshot"
10-08-2017 11:35:40.958 INFO ShutdownHandler - shutting down level "ShutdownLevel
MainThread"
10-08-2017 11:35:40.958 INFO ShutdownHandler - shutting down level "ShutdownLevelExec"
10-08-2017 11:35:40.958 INFO ShutdownHandler - shutting down level "ShutdownLevel
Tailing"
10-08-2017 11:35:40.958 INFO ShutdownHandler - shutting down level "ShutdownLevelPeerManager"
10-08-2017 11:35:40.958 INFO ShutdownHandler - shutting down level "ShutdownLevel
AuditTrailManager"
10-08-2017 11:35:40.958 INFO ShutdownHandler - shutting down level "ShutdownLevelAuditTrailQueueServiceThread"
10-08-2017 11:35:40.958 INFO ShutdownHandler - shutting down level "ShutdownLevel
FSChangeMonitor"
10-08-2017 11:35:40.958 INFO ShutdownHandler - shutting down level "ShutdownLevelFSChangeManagerProcessor"
10-08-2017 11:35:40.958 INFO ShutdownHandler - shutting down level "ShutdownLevel
DeploymentClient"
10-08-2017 11:35:40.958 INFO ShutdownHandler - shutting down level "ShutdownLevelHttpClientPollingThread"
10-08-2017 11:35:40.958 INFO ShutdownHandler - shutting down level "ShutdownLevel
AsyncQueuedMessageDispatcherThread"
10-08-2017 11:35:40.958 INFO ShutdownHandler - shutting down level "ShutdownLevelOfflineFlusher"
10-08-2017 11:35:40.958 INFO ShutdownHandler - shutting down level "ShutdownLevel
Slave"
10-08-2017 11:35:40.958 INFO ShutdownHandler - shutting down level "ShutdownLevelSlaveSearch"
10-08-2017 11:35:40.958 INFO ShutdownHandler - shutting down level "ShutdownLevel
Captain"
10-08-2017 11:35:40.958 INFO ShutdownHandler - shutting down level "ShutdownLevelSelect"
10-08-2017 11:35:40.958 INFO ShutdownHandler - shutting down level "ShutdownLevel
IdataDOCollector"
10-08-2017 11:35:40.958 INFO ShutdownHandler - shutting down level "ShutdownLevel
TcpOutput2"
10-08-2017 11:35:40.958 INFO ShutdownHandler - shutting down level "ShutdownLevelIndexerService"
10-08-2017 11:35:40.958 INFO ShutdownHandler - shutting down level "ShutdownLevel
Database1"
10-08-2017 11:35:40.958 INFO ShutdownHandler - shutting down level "ShutdownLevelLastIndexerLevel"
10-08-2017 11:35:40.958 INFO ShutdownHandler - shutting down level "ShutdownLevel
AWSMetering"
10-08-2017 11:35:40.958 INFO ShutdownHandler - shutting down level "ShutdownLevelTcpInput2"
10-08-2017 11:35:40.958 INFO ShutdownHandler - shutting down level "ShutdownLevel
SearchDispatch"
10-08-2017 11:35:40.958 INFO ShutdownHandler - shutting down level "ShutdownLevelLoadLDAPUsers"
10-08-2017 11:35:40.958 INFO ShutdownHandler - shutting down level "ShutdownLevel
MetricsManager"
10-08-2017 11:35:40.958 INFO ShutdownHandler - shutting down level "ShutdownLevelPipeline"
10-08-2017 11:35:40.958 INFO ShutdownHandler - shutting down level "ShutdownLevel
Queue"
10-08-2017 11:35:40.958 INFO ShutdownHandler - shutting down level "ShutdownLevelCallbackRunner"
10-08-2017 11:35:40.958 INFO ShutdownHandler - shutting down level "ShutdownLevel
HttpClient"
10-08-2017 11:35:40.958 INFO ShutdownHandler - shutting down level "ShutdownLevelDmcProxyHttpClient"
10-08-2017 11:35:40.958 INFO ShutdownHandler - shutting down level "ShutdownLevel
Duo2FAHttpClient"
10-08-2017 11:35:40.958 INFO ShutdownHandler - shutting down level "ShutdownLevelApplicationLicenseChecker"
10-08-2017 11:35:40.958 INFO ShutdownHandler - shutting down level "ShutdownLevel
S3ConnectionPoolManager"
10-08-2017 11:35:40.958 INFO ShutdownHandler - Shutdown complete in 0 microseconds
10-08-2017 11:35:40.959 ERROR dispatchRunner - RunDispatch::runDispatchThread threw error: Error in 'fit' command: Error while initializing algorithm "CorrelationMatrix": Algorithm "CorrelationMatrix" cannot be loaded

thanks for enlightenments!

Andy

0 Karma
Highlighted

Re: Splunk Machine Learning Toolkit: error message when importing new algorithm

Splunk Employee
Splunk Employee

hi @mhyipa - likely the documentation example does not have perfect whitespace.

can you

1.) paste the file contents here in a source block

2.) try using the correlation matrix available in this example repo and see if you have the same problem?
https://github.com/metasyn/mlspl-extensibility-examples

3.) which version of MLTK are you using & which version of PSC (Python for Scientific Computing) ?

0 Karma
Highlighted

Re: Splunk Machine Learning Toolkit: error message when importing new algorithm

Engager

Thanks very much aljohnson!

I downloaded curl and registered the algorithm using REST API, then it works.

Seems that manually editing the existing algos.conf is not working in my case.

My MLTK is 2.4.0 and PSC is 1.2 (64 bit)

Highlighted

Re: Splunk Machine Learning Toolkit: error message when importing new algorithm

Splunk Employee
Splunk Employee

Interesting. I am glad to hear that it is working now!

0 Karma