topic Re: Add libraries to Install Python for Scientific Computing in All Apps and Add-ons

Add libraries to Install Python for Scientific Computing

mhsilva — Wed, 30 Sep 2020 00:16:53 GMT

Hello everyone,
I want to add an algorithm to Splunk, more specifically the Prophet from Facebook.

How can i add modules to Python Scientific Computing, since this module is not available by default?

I tried add this code before importing and use other python from my computer, but i get a lot of errors

import sys
sys.path.append('/home/myuser/anaconda2/lib/python2.7')
sys.path.append('/home/myuser/anaconda2/lib/python2.7/site-package

The error when the search with the algorithm fail:

Error in 'fit' command: Invalid message received from external search command during search, see search.log.

The Job Inspector show this message:

This search did not successfully execute. Any results returned from this job are not consistent and should not be used.

And looking in the search.log through Job Inspector, I found these errors:

04-25-2019 10:06:21.321 ERROR ChunkedExternProcessor - stderr: ERROR:fbprophet.plot:Importing matplotlib failed. Plotting will not work.
04-25-2019 10:06:21.371 ERROR ChunkedExternProcessor - stderr: /home/myuser/.local/lib/python2.7/site-packages/fbprophet/forecaster.py:794: FutureWarning: Series.nonzero() is deprecated and will be removed in a future version.Use Series.to_numpy().nonzero() instead
04-25-2019 10:06:21.371 ERROR ChunkedExternProcessor - stderr: min_dt = dt.iloc[dt.nonzero()[0]].min()
04-25-2019 10:06:21.402 ERROR ChunkedExternProcessor - Failed attempting to parse transport header: Initial log joint probability = -2.68685

Does anyone know how to fix these, or have a new way to add a library?
Thanks

Re: Add libraries to Install Python for Scientific Computing

simonyi — Mon, 10 Feb 2020 12:01:45 GMT

I've just run into the same issue while I wanted to run a 'fit' command with Splunk Machine Learning Toolkit 4.5.0, though I could not reproduce it with MLTK 4.0.0.
It seems that the problem comes up when PYTHONUNBUFFERED environmental variable is set. There is no error if the env var is not set.

In my case stderr stream is handled somehow as transport header:

02-10-2020 10:05:07.837 INFO  ChunkedExternProcessor - Running process: /opt/splunk/bin/python3.7 /opt/splunk/etc/apps/Splunk_ML_Toolkit/bin/fit.py
02-10-2020 10:05:07.870 ERROR ChunkedExternProcessor - Failed attempting to parse transport header: INFO Running /opt/splunk/etc/apps/Splunk_SA_Scientific_Python_linux_x86_64/bin/linux_x86_64/bin/python /opt/splunk/etc/apps/Splunk_ML_Toolkit/bin/fit.py <_io.TextIOWrapper name='<stderr>' mode='w' encoding='UTF-8'>

I guess the INFO part of the printout is supposed to be directed to stderr.

Can you Splunk guys investigate the above one?

Re: Add libraries to Install Python for Scientific Computing

richgalloway — Mon, 10 Feb 2020 13:56:46 GMT

There are no "Splunk guys" here. We're all fellow users. If you need help from Splunk then you should open a support request.

Re: Add libraries to Install Python for Scientific Computing

simonyi — Mon, 10 Feb 2020 14:23:29 GMT

Thanks richgalloway, I was hoping someone from the authors reading this. 🙂

Re: Add libraries to Install Python for Scientific Computing

maraman_splunk — Mon, 10 Feb 2020 14:59:51 GMT

I am not a subject expert on this but if look like the error message say it doesnt find matplotlib library which fbprophet use
As this is reported to stderr , that goes up to the fit command which complain
Also , please make sure you choose the appropriate version of mltk
if the algo you wish add is made for python 3, you probably need to be using mltk 5.0 + splunk 8.0 + adapt the path to not use python 2.x)
If the version are unaligned, that could explain the lib not found error.
What are the versions you are playing with ?

Re: Add libraries to Install Python for Scientific Computing

simonyi — Mon, 10 Feb 2020 15:43:01 GMT

Actually my case is different from the root cause point of view.

I run the following search:

search index="_audit" | fit LinearRegression linecount from _time

So 'fit' fails in case PYTHONUNBUFFERED env var is set and does not fail if it is not set.

Versions are Splunk 8.0.1, MLTK 5.0.0, PSC 2.0.0 with python 3.7.

Re: Add libraries to Install Python for Scientific Computing

garrets — Mon, 26 Aug 2024 09:19:44 GMT

I managed to add extra python modules to Python for Scientific Computing by building it from source.
In my case I added xgboost for Linux (64-bit)
Cloned the GitHub repo: https://github.com/splunk/Splunk-python-for-scientific-computing
Just for increased stability I checked out the latest available git tag (currently 4.2.1, this step might not be necessary)
I then added in environment.nix.yml the python module that I want (in my case: - xgboost=2.1.1)
Afterwards, followed the readme and run: make freeze make build make dist
Finally copied the tarball from the build directory to the user_apps directory on Splunk (substituting the existing Python for Scientific Computing app if already installed).

When using Python for Scientific Computing copy the exec_anaconda and util python scripts to the bin directory of your app. Also copy the lib/splunklib from Python for Scientific Computing to the bin directory of your app.
Add these lines to the start of your script:

import exec_anaconda exec_anaconda.exec_anaconda()