After Splunk upgrade from 9.2.2 to 9.3.1 getting cmath error in saved search

[cadrija]

We upgraded our Splunk enterprise from 9.2.2 to 9.3.1, after the upgrade one of the app is not working as the related saved search is failing to execute.

The saved search is throwing below error.

command="execute", Exception: Traceback (most recent call last): File "/opt/splunk/etc/apps/stormwatch/bin/execute.py", line 16, in <module> from utils.print import print, append_print File "/opt/splunk/etc/apps/stormwatch/bin/utils/print.py", line 3, in <module> import pandas as pd File "/opt/splunk/etc/apps/stormwatch/bin/site-packages/pandas/__init__.py", line 138, in <module> from pandas import testing # noqa:PDF015 File "/opt/splunk/etc/apps/stormwatch/bin/site-packages/pandas/testing.py", line 6, in <module> from pandas._testing import ( File "/opt/splunk/etc/apps/stormwatch/bin/site-packages/pandas/_testing/__init__.py", line 69, in <module> from pandas._testing.asserters import ( File "/opt/splunk/etc/apps/stormwatch/bin/site-packages/pandas/_testing/asserters.py", line 17, in <module> import pandas._libs.testing as _testing File "pandas/_libs/testing.pyx", line 1, in init pandas._libs.testing ModuleNotFoundError: No module named 'cmath'

[livehybrid]

Hi @cadrija 

Is the stormwatch app an internally developed app? It looks like the cmath module is not included in the app - this might have previously been included as part of the previous Splunk version.

You can try installing the cmath library into the site-packages folder using pip using:

/opt/splunk/bin/splunk python3 -m pip install cmath -t /opt/splunk/etc/apps/stormwatch/bin/site-packages

This might temporarily resolve the issue and allow you to determine the best permanent solution, however there is no guarantee that after adding the missing library that there wouldnt be further issues.

Please let me know how you get on and consider adding karma to this or any other answer if it has helped.
Regards

Will

[cadrija]

@livehybrid 
Yes this is an internally developed app.
I tried installing cmath

sudo -H ./splunk cmd python3 -m pip install cmath -t /opt/splunk/etc/apps/stormwatch/bin/site-packages

But getting error 

WARNING: The directory '/root/.cache/pip' or its parent directory is not owned or is not writable by the current user. The cache has been disabled. Check the permissions and owner of that directory. If executing pip with sudo, you should use sudo's -H flag.
ERROR: Could not find a version that satisfies the requirement cmath (from versions: none)
ERROR: No matching distribution found for cmath

[livehybrid]

Actually, it looks like cmath should be a system library - are you adjusting the python lib path in your code, if so, what is it set to?

[gcusello]

Hi @cadrija ,

are you using some python script in your app?

If yes, review your script because python was definitively upgraded to 3.7.

if not, there is anothr issue, and the best solution is to open a case to Splunk Support.

Ciao.

Giuseppe

[cadrija]

@gcusello 
Yes python was upgraded to 3.9 while upgrading Splunk 9.3.1, and it was throwing error to upgrade numpy then I upgraded Numpy to 1.26.0 to make it compatible with the python version.