All Apps and Add-ons

Add libraries to Install Python for Scientific Computing

mhsilva
Explorer

Hello everyone,
I want to add an algorithm to Splunk, more specifically the Prophet from Facebook.

How can i add modules to Python Scientific Computing, since this module is not available by default?

I tried add this code before importing and use other python from my computer, but i get a lot of errors

import sys
sys.path.append('/home/myuser/anaconda2/lib/python2.7')
sys.path.append('/home/myuser/anaconda2/lib/python2.7/site-package

The error when the search with the algorithm fail:

Error in 'fit' command: Invalid message received from external search command during search, see search.log.

The Job Inspector show this message:

This search did not successfully execute. Any results returned from this job are not consistent and should not be used.

And looking in the search.log through Job Inspector, I found these errors:

04-25-2019 10:06:21.321 ERROR ChunkedExternProcessor - stderr: ERROR:fbprophet.plot:Importing matplotlib failed. Plotting will not work.
04-25-2019 10:06:21.371 ERROR ChunkedExternProcessor - stderr: /home/myuser/.local/lib/python2.7/site-packages/fbprophet/forecaster.py:794: FutureWarning: Series.nonzero() is deprecated and will be removed in a future version.Use Series.to_numpy().nonzero() instead
04-25-2019 10:06:21.371 ERROR ChunkedExternProcessor - stderr: min_dt = dt.iloc[dt.nonzero()[0]].min()
04-25-2019 10:06:21.402 ERROR ChunkedExternProcessor - Failed attempting to parse transport header: Initial log joint probability = -2.68685

Does anyone know how to fix these, or have a new way to add a library?
Thanks

0 Karma

maraman_splunk
Splunk Employee
Splunk Employee

I am not a subject expert on this but if look like the error message say it doesnt find matplotlib library which fbprophet use
As this is reported to stderr , that goes up to the fit command which complain
Also , please make sure you choose the appropriate version of mltk
if the algo you wish add is made for python 3, you probably need to be using mltk 5.0 + splunk 8.0 + adapt the path to not use python 2.x)
If the version are unaligned, that could explain the lib not found error.
What are the versions you are playing with ?

0 Karma

simonyi
New Member

Actually my case is different from the root cause point of view.

I run the following search:

search index="_audit" | fit LinearRegression linecount from _time

So 'fit' fails in case PYTHONUNBUFFERED env var is set and does not fail if it is not set.

Versions are Splunk 8.0.1, MLTK 5.0.0, PSC 2.0.0 with python 3.7.

0 Karma

simonyi
New Member

I've just run into the same issue while I wanted to run a 'fit' command with Splunk Machine Learning Toolkit 4.5.0, though I could not reproduce it with MLTK 4.0.0.
It seems that the problem comes up when PYTHONUNBUFFERED environmental variable is set. There is no error if the env var is not set.

In my case stderr stream is handled somehow as transport header:

02-10-2020 10:05:07.837 INFO  ChunkedExternProcessor - Running process: /opt/splunk/bin/python3.7 /opt/splunk/etc/apps/Splunk_ML_Toolkit/bin/fit.py
02-10-2020 10:05:07.870 ERROR ChunkedExternProcessor - Failed attempting to parse transport header: INFO Running /opt/splunk/etc/apps/Splunk_SA_Scientific_Python_linux_x86_64/bin/linux_x86_64/bin/python /opt/splunk/etc/apps/Splunk_ML_Toolkit/bin/fit.py <_io.TextIOWrapper name='<stderr>' mode='w' encoding='UTF-8'>

I guess the INFO part of the printout is supposed to be directed to stderr.

Can you Splunk guys investigate the above one?

0 Karma

richgalloway
SplunkTrust
SplunkTrust

There are no "Splunk guys" here. We're all fellow users. If you need help from Splunk then you should open a support request.

---
If this reply helps you, an upvote would be appreciated.
0 Karma

simonyi
New Member

Thanks richgalloway, I was hoping someone from the authors reading this. 🙂

0 Karma