Updated Splunk Enterprise and Palo Also App crashing

eportenier — Thu, 11 Feb 2021 19:51:32 GMT

We went through an upgrade to the latest version of Slunk Enterprise, no problem. However, when we started to upgrade the apps - specifically a Palo Alto plug-in app - Splunk crashed and now the splunkd service will not stay started. I'm not sure which logs to look at, either, to try to figure out what happened. I have gone through the splunkd.log file but all I see is a bunch of errors (shown below). I know that this isn't much to go on, but can anyone help walk me through trying to troubleshoot this?

Thank you in advance,

Eric

02-11-2021 11:12:26.015 -0800 ERROR PersistentScript - From {"E:\Program Files\Splunk\bin\Python3.exe" "E:\Program Files\Splunk\etc\apps\Splunk_TA_microsoft-cloudservices\bin\splunk_ta_mscs_rh_azureaccount.py" persistent}: from splunk.appserver.mrsparkle.lib.util import make_splunkhome_path
02-11-2021 11:12:26.015 -0800 ERROR PersistentScript - From {"E:\Program Files\Splunk\bin\Python3.exe" "E:\Program Files\Splunk\etc\apps\Splunk_TA_microsoft-cloudservices\bin\splunk_ta_mscs_rh_azureaccount.py" persistent}: File "E:\Program Files\Splunk\Python-3.7\lib\site-packages\splunk\appserver\mrsparkle\lib\util.py", line 30, in <module>
02-11-2021 11:12:26.015 -0800 ERROR PersistentScript - From {"E:\Program Files\Splunk\bin\Python3.exe" "E:\Program Files\Splunk\etc\apps\Splunk_TA_microsoft-cloudservices\bin\splunk_ta_mscs_rh_azureaccount.py" persistent}: import splunk.search.Parser
02-11-2021 11:12:26.015 -0800 ERROR PersistentScript - From {"E:\Program Files\Splunk\bin\Python3.exe" "E:\Program Files\Splunk\etc\apps\Splunk_TA_microsoft-cloudservices\bin\splunk_ta_mscs_rh_azureaccount.py" persistent}: File "E:\Program Files\Splunk\Python-3.7\lib\site-packages\splunk\search\Parser.py", line 7, in <module>
02-11-2021 11:12:26.015 -0800 ERROR PersistentScript - From {"E:\Program Files\Splunk\bin\Python3.exe" "E:\Program Files\Splunk\etc\apps\Splunk_TA_microsoft-cloudservices\bin\splunk_ta_mscs_rh_azureaccount.py" persistent}: import httplib2
02-11-2021 11:12:26.015 -0800 ERROR PersistentScript - From {"E:\Program Files\Splunk\bin\Python3.exe" "E:\Program Files\Splunk\etc\apps\Splunk_TA_microsoft-cloudservices\bin\splunk_ta_mscs_rh_azureaccount.py" persistent}: File "E:\Program Files\Splunk\etc\apps\Splunk_TA_microsoft-cloudservices\bin\splunktamscs\httplib2\__init__.py", line 28, in <module>
02-11-2021 11:12:26.015 -0800 ERROR PersistentScript - From {"E:\Program Files\Splunk\bin\Python3.exe" "E:\Program Files\Splunk\etc\apps\Splunk_TA_microsoft-cloudservices\bin\splunk_ta_mscs_rh_azureaccount.py" persistent}: import email.FeedParser
02-11-2021 11:12:26.015 -0800 ERROR PersistentScript - From {"E:\Program Files\Splunk\bin\Python3.exe" "E:\Program Files\Splunk\etc\apps\Splunk_TA_microsoft-cloudservices\bin\splunk_ta_mscs_rh_azureaccount.py" persistent}: ModuleNotFoundError: No module named 'email.FeedParser'
02-11-2021 11:12:26.513 -0800 WARN PersistentScript - Process {"E:\Program Files\Splunk\bin\Python3.exe" "E:\Program Files\Splunk\etc\apps\Splunk_TA_microsoft-cloudservices\bin\splunk_ta_mscs_rh_azureaccount.py" persistent}: exited with code 1
02-11-2021 11:12:27.014 -0800 ERROR PersistentScript - From {"E:\Program Files\Splunk\bin\Python3.exe" "E:\Program Files\Splunk\etc\apps\Splunk_TA_microsoft-cloudservices\bin\splunk_ta_mscs_rh_azureaccount.py" persistent}: Traceback (most recent call last):
02-11-2021 11:12:27.014 -0800 ERROR AdminManagerExternal - Received malformed XML from external handler:
02-11-2021 11:12:27.014 -0800 ERROR AdminManagerExternal - Unable to xml-parse the following data: %s
02-11-2021 11:12:27.014 -0800 ERROR PersistentScript - From {"E:\Program Files\Splunk\bin\Python3.exe" "E:\Program Files\Splunk\etc\apps\Splunk_TA_microsoft-cloudservices\bin\splunk_ta_mscs_rh_azureaccount.py" persistent}: File "E:\Program Files\Splunk\etc\apps\Splunk_TA_microsoft-cloudservices\bin\splunk_ta_mscs_rh_azureaccount.py", line 19, in <module>
02-11-2021 11:12:27.014 -0800 ERROR PersistentScript - From {"E:\Program Files\Splunk\bin\Python3.exe" "E:\Program Files\Splunk\etc\apps\Splunk_TA_microsoft-cloudservices\bin\splunk_ta_mscs_rh_azureaccount.py" persistent}: from mscs_util import mscs_consts, get_proxy_info_from_endpoint, get_logger, check_account_isvalid
02-11-2021 11:12:27.014 -0800 ERROR PersistentScript - From {"E:\Program Files\Splunk\bin\Python3.exe" "E:\Program Files\Splunk\etc\apps\Splunk_TA_microsoft-cloudservices\bin\splunk_ta_mscs_rh_azureaccount.py" persistent}: File "E:\Program Files\Splunk\etc\apps\Splunk_TA_microsoft-cloudservices\bin\splunktamscs\mscs_util.py", line 13, in <module>
02-11-2021 11:12:27.014 -0800 ERROR PersistentScript - From {"E:\Program Files\Splunk\bin\Python3.exe" "E:\Program Files\Splunk\etc\apps\Splunk_TA_microsoft-cloudservices\bin\splunk_ta_mscs_rh_azureaccount.py" persistent}: from splunk.appserver.mrsparkle.lib.util import make_splunkhome_path
02-11-2021 11:12:27.014 -0800 ERROR PersistentScript - From {"E:\Program Files\Splunk\bin\Python3.exe" "E:\Program Files\Splunk\etc\apps\Splunk_TA_microsoft-cloudservices\bin\splunk_ta_mscs_rh_azureaccount.py" persistent}: File "E:\Program Files\Splunk\Python-3.7\lib\site-packages\splunk\appserver\mrsparkle\lib\util.py", line 30, in <module>
02-11-2021 11:12:27.014 -0800 ERROR PersistentScript - From {"E:\Program Files\Splunk\bin\Python3.exe" "E:\Program Files\Splunk\etc\apps\Splunk_TA_microsoft-cloudservices\bin\splunk_ta_mscs_rh_azureaccount.py" persistent}: import splunk.search.Parser
02-11-2021 11:12:27.014 -0800 ERROR PersistentScript - From {"E:\Program Files\Splunk\bin\Python3.exe" "E:\Program Files\Splunk\etc\apps\Splunk_TA_microsoft-cloudservices\bin\splunk_ta_mscs_rh_azureaccount.py" persistent}: File "E:\Program Files\Splunk\Python-3.7\lib\site-packages\splunk\search\Parser.py", line 7, in <module>
02-11-2021 11:12:27.014 -0800 ERROR PersistentScript - From {"E:\Program Files\Splunk\bin\Python3.exe" "E:\Program Files\Splunk\etc\apps\Splunk_TA_microsoft-cloudservices\bin\splunk_ta_mscs_rh_azureaccount.py" persistent}: import httplib2
02-11-2021 11:12:27.014 -0800 ERROR PersistentScript - From {"E:\Program Files\Splunk\bin\Python3.exe" "E:\Program Files\Splunk\etc\apps\Splunk_TA_microsoft-cloudservices\bin\splunk_ta_mscs_rh_azureaccount.py" persistent}: File "E:\Program Files\Splunk\etc\apps\Splunk_TA_microsoft-cloudservices\bin\splunktamscs\httplib2\__init__.py", line 28, in <module>
02-11-2021 11:12:27.014 -0800 ERROR PersistentScript - From {"E:\Program Files\Splunk\bin\Python3.exe" "E:\Program Files\Splunk\etc\apps\Splunk_TA_microsoft-cloudservices\bin\splunk_ta_mscs_rh_azureaccount.py" persistent}: import email.FeedParser
02-11-2021 11:12:27.014 -0800 ERROR PersistentScript - From {"E:\Program Files\Splunk\bin\Python3.exe" "E:\Program Files\Splunk\etc\apps\Splunk_TA_microsoft-cloudservices\bin\splunk_ta_mscs_rh_azureaccount.py" persistent}: ModuleNotFoundError: No module named 'email.FeedParser'
02-11-2021 11:12:27.014 -0800 WARN PersistentScript - Process {"E:\Program Files\Splunk\bin\Python3.exe" "E:\Program Files\Splunk\etc\apps\Splunk_TA_microsoft-cloudservices\bin\splunk_ta_mscs_rh_azureaccount.py" persistent}: exited with code 1
02-11-2021 11:12:27.051 -0800 ERROR AdminManagerExternal - Received malformed XML from external handler:
02-11-2021 11:12:27.051 -0800 ERROR AdminManagerExternal - Unable to xml-parse the following data: %s
02-11-2021 11:12:29.073 -0800 WARN PersistentScript - Process {"E:\Program Files\Splunk\bin\Python3.exe" "E:\Program Files\Splunk\etc\apps\Splunk_TA_microsoft-cloudservices\bin\splunk_ta_mscs_rh_azureaccount.py" persistent}: exited with code 1
02-11-2021 11:12:29.574 -0800 ERROR PersistentScript - From {"E:\Program Files\Splunk\bin\Python3.exe" "E:\Program Files\Splunk\etc\apps\Splunk_TA_microsoft-cloudservices\bin\splunk_ta_mscs_rh_azureaccount.py" persistent}: Traceback (most recent call last):
02-11-2021 11:12:29.574 -0800 ERROR PersistentScript - From {"E:\Program Files\Splunk\bin\Python3.exe" "E:\Program Files\Splunk\etc\apps\Splunk_TA_microsoft-cloudservices\bin\splunk_ta_mscs_rh_azureaccount.py" persistent}: File "E:\Program Files\Splunk\etc\apps\Splunk_TA_microsoft-cloudservices\bin\splunk_ta_mscs_rh_azureaccount.py", line 19, in <module>
02-11-2021 11:12:29.574 -0800 ERROR AdminManagerExternal - Received malformed XML from external handler:
02-11-2021 11:12:29.574 -0800 ERROR AdminManagerExternal - Unable to xml-parse the following data: %s
02-11-2021 11:12:29.574 -0800 ERROR PersistentScript - From {"E:\Program Files\Splunk\bin\Python3.exe" "E:\Program Files\Splunk\etc\apps\Splunk_TA_microsoft-cloudservices\bin\splunk_ta_mscs_rh_azureaccount.py" persistent}: from mscs_util import mscs_consts, get_proxy_info_from_endpoint, get_logger, check_account_isvalid
02-11-2021 11:12:29.574 -0800 ERROR PersistentScript - From {"E:\Program Files\Splunk\bin\Python3.exe" "E:\Program Files\Splunk\etc\apps\Splunk_TA_microsoft-cloudservices\bin\splunk_ta_mscs_rh_azureaccount.py" persistent}: File "E:\Program Files\Splunk\etc\apps\Splunk_TA_microsoft-cloudservices\bin\splunktamscs\mscs_util.py", line 13, in <module>
02-11-2021 11:12:29.574 -0800 ERROR PersistentScript - From {"E:\Program Files\Splunk\bin\Python3.exe" "E:\Program Files\Splunk\etc\apps\Splunk_TA_microsoft-cloudservices\bin\splunk_ta_mscs_rh_azureaccount.py" persistent}: from splunk.appserver.mrsparkle.lib.util import make_splunkhome_path
02-11-2021 11:12:29.574 -0800 ERROR PersistentScript - From {"E:\Program Files\Splunk\bin\Python3.exe" "E:\Program Files\Splunk\etc\apps\Splunk_TA_microsoft-cloudservices\bin\splunk_ta_mscs_rh_azureaccount.py" persistent}: File "E:\Program Files\Splunk\Python-3.7\lib\site-packages\splunk\appserver\mrsparkle\lib\util.py", line 30, in <module>
02-11-2021 11:12:29.574 -0800 ERROR PersistentScript - From {"E:\Program Files\Splunk\bin\Python3.exe" "E:\Program Files\Splunk\etc\apps\Splunk_TA_microsoft-cloudservices\bin\splunk_ta_mscs_rh_azureaccount.py" persistent}: import splunk.search.Parser
02-11-2021 11:12:29.574 -0800 ERROR PersistentScript - From {"E:\Program Files\Splunk\bin\Python3.exe" "E:\Program Files\Splunk\etc\apps\Splunk_TA_microsoft-cloudservices\bin\splunk_ta_mscs_rh_azureaccount.py" persistent}: File "E:\Program Files\Splunk\Python-3.7\lib\site-packages\splunk\search\Parser.py", line 7, in <module>
02-11-2021 11:12:29.574 -0800 ERROR PersistentScript - From {"E:\Program Files\Splunk\bin\Python3.exe" "E:\Program Files\Splunk\etc\apps\Splunk_TA_microsoft-cloudservices\bin\splunk_ta_mscs_rh_azureaccount.py" persistent}: import httplib2
02-11-2021 11:12:29.574 -0800 ERROR PersistentScript - From {"E:\Program Files\Splunk\bin\Python3.exe" "E:\Program Files\Splunk\etc\apps\Splunk_TA_microsoft-cloudservices\bin\splunk_ta_mscs_rh_azureaccount.py" persistent}: File "E:\Program Files\Splunk\etc\apps\Splunk_TA_microsoft-cloudservices\bin\splunktamscs\httplib2\__init__.py", line 28, in <module>
02-11-2021 11:12:29.574 -0800 ERROR PersistentScript - From {"E:\Program Files\Splunk\bin\Python3.exe" "E:\Program Files\Splunk\etc\apps\Splunk_TA_microsoft-cloudservices\bin\splunk_ta_mscs_rh_azureaccount.py" persistent}: import email.FeedParser
02-11-2021 11:12:29.574 -0800 ERROR PersistentScript - From {"E:\Program Files\Splunk\bin\Python3.exe" "E:\Program Files\Splunk\etc\apps\Splunk_TA_microsoft-cloudservices\bin\splunk_ta_mscs_rh_azureaccount.py" persistent}: ModuleNotFoundError: No module named 'email.FeedParser'
02-11-2021 11:12:30.629 -0800 INFO ClientSessionsManager - Adding client: ip=10. uts=windows-x64 id=53CDFDC2-FA78-4D51-B339-B52DFC28F996 name=53CDFDC2-FA78-4D51-B339-B52DFC28F996

Re: Updated Splunk Enterprise and Palo Also App crashing

scelikok — Thu, 11 Feb 2021 20:01:52 GMT

Hi @eportenier,

It seems there is an Python3 incompatibility on Splunk_TA_microsoft-cloudservices app. SplunkWeb cannot start. Can you please try disabling this app and restart?

To prevent issues starting Splunk Web, revise apps for Python 3 compatibility. If an app cannot be upgraded, it must be removed for Splunk Web to start.

https://docs.splunk.com/Documentation/Splunk/8.1.2/Python3Migration/ChangesEnterprise#Splunk_Web

Re: Updated Splunk Enterprise and Palo Also App crashing

eportenier — Thu, 11 Feb 2021 20:27:24 GMT

Thank you for your reply! I hate sounding stupid, but where do I disable an app from starting? I'm assuming that there is a configuration file somewhere? Sorry for the dumb question... I know I should probably know this but I haven't really had to delve into the inner workings of Splunk lately.

Re: Updated Splunk Enterprise and Palo Also App crashing

eportenier — Thu, 11 Feb 2021 20:50:26 GMT

That's what it was. 🙂 I was able to find the App folder that had the installed apps in it. I removed the "bad" app and started the service... it came back! Thank you very much for your assistance!!

Eric

Re: Updated Splunk Enterprise and Palo Also App crashing

isoutamo — Thu, 11 Feb 2021 21:45:50 GMT

before update to 8.x.x you should check readiness of apps with https://splunkbase.splunk.com/app/4698/

r. Ismo

topic Updated Splunk Enterprise and Palo Also App crashing in Splunk Enterprise

Updated Splunk Enterprise and Palo Also App crashing

Re: Updated Splunk Enterprise and Palo Also App crashing

Re: Updated Splunk Enterprise and Palo Also App crashing

Re: Updated Splunk Enterprise and Palo Also App crashing

Re: Updated Splunk Enterprise and Palo Also App crashing