topic Re: Splunk Machine Learning Tool kit: Error in 'fit' command: External search command exited unexpectedly. in All Apps and Add-ons

Where to fix Splunk Machine Learning Tool kit: Error in 'fit' command: External search command exited unexpectedly.?

arlombar1 — Mon, 08 May 2023 14:41:12 GMT

Hello,

I just installed the ML toolkit for Splunk and am running into the below error every time I go to the showcase and attempt to use one of the dashboards and create a "fit model".

Error in 'fit' command: External search command exited unexpectedly.

I've also tested the SPL search using the fit command and I get the same error.

I've made sure to download Python for Scientific Computing Add-on specific to my OS (linux) prior to installing the ML toolkit and made sure that the permissions are all global.

Splunkd.log is only show the below error message:
07-26-2019 14:39:01.121 -0400 ERROR ChunkedExternProcessor - EOF while attempting to read transport header
07-26-2019 14:39:01.121 -0400 ERROR ChunkedExternProcessor - Error in 'fit' command: External search command exited unexpectedly.

Splunk version: 7.0.2

Has anyone run into this issue before with this app, or maybe a different app with the same error? The fit command is essentially a python script and I've been combing through the code, but can't determine which line the EOF error is referencing.

Re: Splunk Machine Learning Tool kit: Error in 'fit' command: External search command exited unexpectedly.

BigCosta — Wed, 30 Sep 2020 02:36:15 GMT

Hello!
I have the same problem. I researched the problem and found that the problem is in the numpy module.
The code of Python "import numpy as np" causes "Illegal instruction (core dumped)"
You can check this by running the command:
/opt/splunk/etc/apps/Splunk_SA_Scientific_Python_linux_x86_64/bin/linux_x86_64/bin/python -c "import numpy as np"

You will see a lot of errors like "ERROR:root:code for hash xxxxx was not found." - that's okay.
If you have no problems with the "fit" command, then the last lines of the output will be:

ValueError: unsupported hash type sha512

If you have problems with the "fit" command, then the last lines of the output will be:

ValueError: unsupported hash type sha512
Illegal instruction (core dumped)

I have not yet found why the numpy module does not start...
I have Splunk 7.3.2, Splunk_SA_Scientific_Python_linux_x86_64 1.4 and Splunk_ML_Toolkit 4.4.1

Re: Splunk Machine Learning Tool kit: Error in 'fit' command: External search command exited unexpectedly.

BigCosta — Wed, 30 Sep 2020 02:36:27 GMT

Hi!
I found a solution to this problem. You need to update three Python packages that come as part of Splunk_SA_Scientific_Python_linux_x86_64 -- NumPy, SciPy and scikit_learn
To do this You need to do the following:

Download from Python software repository https://pypi.org/ latest numpy, scipy and scikit_learn packages (yum won't help because it downloads older versions)

numpy-1.16.5-cp27-cp27mu-manylinux1_x86_64.whl
scipy-1.2.2-cp27-cp27mu-manylinux1_x86_64.whl
scikit_learn-0.20.4-cp27-cp27mu-manylinux1_x86_64.whl
Install packages in system python

pip install numpy-1.16.5-cp27-cp27mu-manylinux1_x86_64.whl
pip install scipy-1.2.2-cp27-cp27mu-manylinux1_x86_64.whl
pip install scikit_learn-0.20.4-cp27-cp27mu-manylinux1_x86_64.whl
Make a backup of the original numpy, scipy and scikit_learn packages located in Splunk_SA_Scientific_Python_linux_x86_64

mv $SPLUNK_HOME/etc/apps/Splunk_SA_Scientific_Python_linux_x86_64/bin/linux_x86_64/lib/python2.7/site-packages/numpy $SPLUNK_HOME/etc/apps/Splunk_SA_Scientific_Python_linux_x86_64/bin/linux_x86_64/lib/python2.7/site-packages/numpy.orig
mv $SPLUNK_HOME/etc/apps/Splunk_SA_Scientific_Python_linux_x86_64/bin/linux_x86_64/lib/python2.7/site-packages/scipy $SPLUNK_HOME/etc/apps/Splunk_SA_Scientific_Python_linux_x86_64/bin/linux_x86_64/lib/python2.7/site-packages/scipy.orig
mv $SPLUNK_HOME/etc/apps/Splunk_SA_Scientific_Python_linux_x86_64/bin/linux_x86_64/lib/python2.7/site-packages/sklearn $SPLUNK_HOME/etc/apps/Splunk_SA_Scientific_Python_linux_x86_64/bin/linux_x86_64/lib/python2.7/site-packages/sklearn.orig
Copy installed packages from system python to Splunk_SA_Scientific_Python_linux_x86_64

cp -r /usr/lib64/python2.7/site-packages/numpy* -t $SPLUNK_HOME/etc/apps/Splunk_SA_Scientific_Python_linux_x86_64/bin/linux_x86_64/lib/python2.7/site-packages
cp -r /usr/lib64/python2.7/site-packages/scipy* -t $SPLUNK_HOME/etc/apps/Splunk_SA_Scientific_Python_linux_x86_64/bin/linux_x86_64/lib/python2.7/site-packages
cp -r /usr/lib64/python2.7/site-packages/sklearn -t $SPLUNK_HOME/etc/apps/Splunk_SA_Scientific_Python_linux_x86_64/bin/linux_x86_64/lib/python2.7/site-packages
All done, you can now run the search with the fit command (Splunk reboot is not required)

The numpy, scipy and scikit_learn packages in system python are no longer needed and you can remove them with the pip uninstall <package> command

Re: Splunk Machine Learning Tool kit: Error in 'fit' command: External search command exited unexpectedly.

fsaporito — Tue, 12 Nov 2019 10:23:20 GMT

This happens also with Splunk 8 and python 3, but I do not tried this fix yet.

Re: Splunk Machine Learning Tool kit: Error in 'fit' command: External search command exited unexpectedly.

eca — Fri, 18 Sep 2020 13:31:41 GMT

Hello @BigCosta,

I have a problem similar to yours, were you able to address it?

Thanks

Best Regards

Emiliano

Re: Splunk Machine Learning Tool kit: Error in 'fit' command: External search command exited unexpectedly.

BigCosta — Mon, 21 Sep 2020 01:46:52 GMT

Hi Emiliano!

Yes, i solved this problem.

Solution in my post in this thread from date 10-12-2019 08:41 PM.

Re: Splunk Machine Learning Tool kit: Error in 'fit' command: External search command exited unexpectedly.

BernardEAI — Wed, 04 Nov 2020 12:03:27 GMT

Hi BigCosta

I am also getting the "Error in 'fit' command: External search command exited unexpectedly" error. I tried running python -c "import numpy as np" in /opt/splunk/etc/apps/Splunk_SA_Scientific_Python_linux_x86_64/bin/linux_x86_64/bin, but I got the error message ImportError: No module named numpy. Does this mean we don't even have numpy installed?

We are running Splunk 7.3.3, Splunk Machine Learning Toolkit 4.5.0

Re: Splunk Machine Learning Tool kit: Error in 'fit' command: External search command exited unexpectedly.

eca — Wed, 04 Nov 2020 13:16:28 GMT

Hi @BernardEAI ,

I solve the problem after install PSC 1.3 and fix some libraries problem on my Linux. On my Linux if I try to execute the command "openssl" I get some errors due to libraries not found. I fix the problem creating sym links to the right library with the right name. The version of MLTK is 4.5 also for me with Splunk 7.3.4.

Hope this help

Best Regards

Emiliano

Re: Splunk Machine Learning Tool kit: Error in 'fit' command: External search command exited unexpectedly.

BernardEAI — Fri, 27 Nov 2020 13:58:34 GMT

Hi @eca

Thanks for the tip on the PSC 1.3 - I installed this version and it solved the problem (without any other changes)! I'm very confused about why this is, and why there isn't better documentation on this.

Our current setup: Splunk Version:7.3.3, MLTK 4.2.0 and PSC 1.3.

Re: Splunk Machine Learning Tool kit: Error in 'fit' command: External search command exited unexpectedly.

splunkoptimus — Mon, 08 May 2023 13:40:54 GMT

I had a similar problem, I resolved it by trying of the different versions of Splunk_SA_Scientific_Python which were supposed to be compatible with the version of Splunk MLTK I was using.

Splunk MLTK v540 is fully compatible to Splunk SA Scientific Python linux x86 64 v310