I just installed the ML toolkit for Splunk and am running into the below error every time I go to the showcase and attempt to use one of the dashboards and create a "fit model".
Error in 'fit' command: External search command exited unexpectedly.
I've also tested the SPL search using the fit command and I get the same error.
I've made sure to download Python for Scientific Computing Add-on specific to my OS (linux) prior to installing the ML toolkit and made sure that the permissions are all global.
Splunkd.log is only show the below error message:
07-26-2019 14:39:01.121 -0400 ERROR ChunkedExternProcessor - EOF while attempting to read transport header
07-26-2019 14:39:01.121 -0400 ERROR ChunkedExternProcessor - Error in 'fit' command: External search command exited unexpectedly.
Splunk version: 7.0.2
Has anyone run into this issue before with this app, or maybe a different app with the same error? The fit command is essentially a python script and I've been combing through the code, but can't determine which line the EOF error is referencing.
I found a solution to this problem. You need to update three Python packages that come as part of Splunk_SA_Scientific_Python_linux_x86_64 -- NumPy, SciPy and scikit_learn
To do this You need to do the following:
Download from Python software repository https://pypi.org/ latest numpy, scipy and scikit_learn packages (yum won't help because it downloads older versions)
Install packages in system python
pip install numpy-1.16.5-cp27-cp27mu-manylinux1_x86_64.whl
pip install scipy-1.2.2-cp27-cp27mu-manylinux1_x86_64.whl
pip install scikit_learn-0.20.4-cp27-cp27mu-manylinux1_x86_64.whl
Make a backup of the original numpy, scipy and scikit_learn packages located in Splunk_SA_Scientific_Python_linux_x86_64
mv $SPLUNK_HOME/etc/apps/Splunk_SA_Scientific_Python_linux_x86_64/bin/linux_x86_64/lib/python2.7/site-packages/numpy $SPLUNK_HOME/etc/apps/Splunk_SA_Scientific_Python_linux_x86_64/bin/linux_x86_64/lib/python2.7/site-packages/numpy.orig
mv $SPLUNK_HOME/etc/apps/Splunk_SA_Scientific_Python_linux_x86_64/bin/linux_x86_64/lib/python2.7/site-packages/scipy $SPLUNK_HOME/etc/apps/Splunk_SA_Scientific_Python_linux_x86_64/bin/linux_x86_64/lib/python2.7/site-packages/scipy.orig
mv $SPLUNK_HOME/etc/apps/Splunk_SA_Scientific_Python_linux_x86_64/bin/linux_x86_64/lib/python2.7/site-packages/sklearn $SPLUNK_HOME/etc/apps/Splunk_SA_Scientific_Python_linux_x86_64/bin/linux_x86_64/lib/python2.7/site-packages/sklearn.orig
Copy installed packages from system python to Splunk_SA_Scientific_Python_linux_x86_64
cp -r /usr/lib64/python2.7/site-packages/numpy* -t $SPLUNK_HOME/etc/apps/Splunk_SA_Scientific_Python_linux_x86_64/bin/linux_x86_64/lib/python2.7/site-packages
cp -r /usr/lib64/python2.7/site-packages/scipy* -t $SPLUNK_HOME/etc/apps/Splunk_SA_Scientific_Python_linux_x86_64/bin/linux_x86_64/lib/python2.7/site-packages
cp -r /usr/lib64/python2.7/site-packages/sklearn -t $SPLUNK_HOME/etc/apps/Splunk_SA_Scientific_Python_linux_x86_64/bin/linux_x86_64/lib/python2.7/site-packages
All done, you can now run the search with the
fit command (Splunk reboot is not required)
The numpy, scipy and scikit_learn packages in system python are no longer needed and you can remove them with the
pip uninstall <package> command
I have the same problem. I researched the problem and found that the problem is in the numpy module.
The code of Python "
import numpy as np" causes "Illegal instruction (core dumped)"
You can check this by running the command:
/opt/splunk/etc/apps/Splunk_SA_Scientific_Python_linux_x86_64/bin/linux_x86_64/bin/python -c "import numpy as np"
You will see a lot of errors like "ERROR:root:code for hash xxxxx was not found." - that's okay.
If you have no problems with the "fit" command, then the last lines of the output will be:
ValueError: unsupported hash type sha512
If you have problems with the "fit" command, then the last lines of the output will be:
ValueError: unsupported hash type sha512 Illegal instruction (core dumped)
I have not yet found why the numpy module does not start...
I have Splunk 7.3.2, Splunk_SA_Scientific_Python_linux_x86_64 1.4 and Splunk_ML_Toolkit 4.4.1