We went through an upgrade to the latest version of Slunk Enterprise, no problem. However, when we started to upgrade the apps - specifically a Palo Alto plug-in app - Splunk crashed and now the splunkd service will not stay started. I'm not sure which logs to look at, either, to try to figure out what happened. I have gone through the splunkd.log file but all I see is a bunch of errors (shown below). I know that this isn't much to go on, but can anyone help walk me through trying to troubleshoot this?
Thank you in advance,
Eric
02-11-2021 11:12:26.015 -0800 ERROR PersistentScript - From {"E:\Program Files\Splunk\bin\Python3.exe" "E:\Program Files\Splunk\etc\apps\Splunk_TA_microsoft-cloudservices\bin\splunk_ta_mscs_rh_azureaccount.py" persistent}: from splunk.appserver.mrsparkle.lib.util import make_splunkhome_path 02-11-2021 11:12:26.015 -0800 ERROR PersistentScript - From {"E:\Program Files\Splunk\bin\Python3.exe" "E:\Program Files\Splunk\etc\apps\Splunk_TA_microsoft-cloudservices\bin\splunk_ta_mscs_rh_azureaccount.py" persistent}: File "E:\Program Files\Splunk\Python-3.7\lib\site-packages\splunk\appserver\mrsparkle\lib\util.py", line 30, in <module> 02-11-2021 11:12:26.015 -0800 ERROR PersistentScript - From {"E:\Program Files\Splunk\bin\Python3.exe" "E:\Program Files\Splunk\etc\apps\Splunk_TA_microsoft-cloudservices\bin\splunk_ta_mscs_rh_azureaccount.py" persistent}: import splunk.search.Parser 02-11-2021 11:12:26.015 -0800 ERROR PersistentScript - From {"E:\Program Files\Splunk\bin\Python3.exe" "E:\Program Files\Splunk\etc\apps\Splunk_TA_microsoft-cloudservices\bin\splunk_ta_mscs_rh_azureaccount.py" persistent}: File "E:\Program Files\Splunk\Python-3.7\lib\site-packages\splunk\search\Parser.py", line 7, in <module> 02-11-2021 11:12:26.015 -0800 ERROR PersistentScript - From {"E:\Program Files\Splunk\bin\Python3.exe" "E:\Program Files\Splunk\etc\apps\Splunk_TA_microsoft-cloudservices\bin\splunk_ta_mscs_rh_azureaccount.py" persistent}: import httplib2 02-11-2021 11:12:26.015 -0800 ERROR PersistentScript - From {"E:\Program Files\Splunk\bin\Python3.exe" "E:\Program Files\Splunk\etc\apps\Splunk_TA_microsoft-cloudservices\bin\splunk_ta_mscs_rh_azureaccount.py" persistent}: File "E:\Program Files\Splunk\etc\apps\Splunk_TA_microsoft-cloudservices\bin\splunktamscs\httplib2\__init__.py", line 28, in <module> 02-11-2021 11:12:26.015 -0800 ERROR PersistentScript - From {"E:\Program Files\Splunk\bin\Python3.exe" "E:\Program Files\Splunk\etc\apps\Splunk_TA_microsoft-cloudservices\bin\splunk_ta_mscs_rh_azureaccount.py" persistent}: import email.FeedParser 02-11-2021 11:12:26.015 -0800 ERROR PersistentScript - From {"E:\Program Files\Splunk\bin\Python3.exe" "E:\Program Files\Splunk\etc\apps\Splunk_TA_microsoft-cloudservices\bin\splunk_ta_mscs_rh_azureaccount.py" persistent}: ModuleNotFoundError: No module named 'email.FeedParser' 02-11-2021 11:12:26.513 -0800 WARN PersistentScript - Process {"E:\Program Files\Splunk\bin\Python3.exe" "E:\Program Files\Splunk\etc\apps\Splunk_TA_microsoft-cloudservices\bin\splunk_ta_mscs_rh_azureaccount.py" persistent}: exited with code 1 02-11-2021 11:12:27.014 -0800 ERROR PersistentScript - From {"E:\Program Files\Splunk\bin\Python3.exe" "E:\Program Files\Splunk\etc\apps\Splunk_TA_microsoft-cloudservices\bin\splunk_ta_mscs_rh_azureaccount.py" persistent}: Traceback (most recent call last): 02-11-2021 11:12:27.014 -0800 ERROR AdminManagerExternal - Received malformed XML from external handler: 02-11-2021 11:12:27.014 -0800 ERROR AdminManagerExternal - Unable to xml-parse the following data: %s 02-11-2021 11:12:27.014 -0800 ERROR PersistentScript - From {"E:\Program Files\Splunk\bin\Python3.exe" "E:\Program Files\Splunk\etc\apps\Splunk_TA_microsoft-cloudservices\bin\splunk_ta_mscs_rh_azureaccount.py" persistent}: File "E:\Program Files\Splunk\etc\apps\Splunk_TA_microsoft-cloudservices\bin\splunk_ta_mscs_rh_azureaccount.py", line 19, in <module> 02-11-2021 11:12:27.014 -0800 ERROR PersistentScript - From {"E:\Program Files\Splunk\bin\Python3.exe" "E:\Program Files\Splunk\etc\apps\Splunk_TA_microsoft-cloudservices\bin\splunk_ta_mscs_rh_azureaccount.py" persistent}: from mscs_util import mscs_consts, get_proxy_info_from_endpoint, get_logger, check_account_isvalid 02-11-2021 11:12:27.014 -0800 ERROR PersistentScript - From {"E:\Program Files\Splunk\bin\Python3.exe" "E:\Program Files\Splunk\etc\apps\Splunk_TA_microsoft-cloudservices\bin\splunk_ta_mscs_rh_azureaccount.py" persistent}: File "E:\Program Files\Splunk\etc\apps\Splunk_TA_microsoft-cloudservices\bin\splunktamscs\mscs_util.py", line 13, in <module> 02-11-2021 11:12:27.014 -0800 ERROR PersistentScript - From {"E:\Program Files\Splunk\bin\Python3.exe" "E:\Program Files\Splunk\etc\apps\Splunk_TA_microsoft-cloudservices\bin\splunk_ta_mscs_rh_azureaccount.py" persistent}: from splunk.appserver.mrsparkle.lib.util import make_splunkhome_path 02-11-2021 11:12:27.014 -0800 ERROR PersistentScript - From {"E:\Program Files\Splunk\bin\Python3.exe" "E:\Program Files\Splunk\etc\apps\Splunk_TA_microsoft-cloudservices\bin\splunk_ta_mscs_rh_azureaccount.py" persistent}: File "E:\Program Files\Splunk\Python-3.7\lib\site-packages\splunk\appserver\mrsparkle\lib\util.py", line 30, in <module> 02-11-2021 11:12:27.014 -0800 ERROR PersistentScript - From {"E:\Program Files\Splunk\bin\Python3.exe" "E:\Program Files\Splunk\etc\apps\Splunk_TA_microsoft-cloudservices\bin\splunk_ta_mscs_rh_azureaccount.py" persistent}: import splunk.search.Parser 02-11-2021 11:12:27.014 -0800 ERROR PersistentScript - From {"E:\Program Files\Splunk\bin\Python3.exe" "E:\Program Files\Splunk\etc\apps\Splunk_TA_microsoft-cloudservices\bin\splunk_ta_mscs_rh_azureaccount.py" persistent}: File "E:\Program Files\Splunk\Python-3.7\lib\site-packages\splunk\search\Parser.py", line 7, in <module> 02-11-2021 11:12:27.014 -0800 ERROR PersistentScript - From {"E:\Program Files\Splunk\bin\Python3.exe" "E:\Program Files\Splunk\etc\apps\Splunk_TA_microsoft-cloudservices\bin\splunk_ta_mscs_rh_azureaccount.py" persistent}: import httplib2 02-11-2021 11:12:27.014 -0800 ERROR PersistentScript - From {"E:\Program Files\Splunk\bin\Python3.exe" "E:\Program Files\Splunk\etc\apps\Splunk_TA_microsoft-cloudservices\bin\splunk_ta_mscs_rh_azureaccount.py" persistent}: File "E:\Program Files\Splunk\etc\apps\Splunk_TA_microsoft-cloudservices\bin\splunktamscs\httplib2\__init__.py", line 28, in <module> 02-11-2021 11:12:27.014 -0800 ERROR PersistentScript - From {"E:\Program Files\Splunk\bin\Python3.exe" "E:\Program Files\Splunk\etc\apps\Splunk_TA_microsoft-cloudservices\bin\splunk_ta_mscs_rh_azureaccount.py" persistent}: import email.FeedParser 02-11-2021 11:12:27.014 -0800 ERROR PersistentScript - From {"E:\Program Files\Splunk\bin\Python3.exe" "E:\Program Files\Splunk\etc\apps\Splunk_TA_microsoft-cloudservices\bin\splunk_ta_mscs_rh_azureaccount.py" persistent}: ModuleNotFoundError: No module named 'email.FeedParser' 02-11-2021 11:12:27.014 -0800 WARN PersistentScript - Process {"E:\Program Files\Splunk\bin\Python3.exe" "E:\Program Files\Splunk\etc\apps\Splunk_TA_microsoft-cloudservices\bin\splunk_ta_mscs_rh_azureaccount.py" persistent}: exited with code 1 02-11-2021 11:12:27.051 -0800 ERROR AdminManagerExternal - Received malformed XML from external handler: 02-11-2021 11:12:27.051 -0800 ERROR AdminManagerExternal - Unable to xml-parse the following data: %s 02-11-2021 11:12:29.073 -0800 WARN PersistentScript - Process {"E:\Program Files\Splunk\bin\Python3.exe" "E:\Program Files\Splunk\etc\apps\Splunk_TA_microsoft-cloudservices\bin\splunk_ta_mscs_rh_azureaccount.py" persistent}: exited with code 1 02-11-2021 11:12:29.574 -0800 ERROR PersistentScript - From {"E:\Program Files\Splunk\bin\Python3.exe" "E:\Program Files\Splunk\etc\apps\Splunk_TA_microsoft-cloudservices\bin\splunk_ta_mscs_rh_azureaccount.py" persistent}: Traceback (most recent call last): 02-11-2021 11:12:29.574 -0800 ERROR PersistentScript - From {"E:\Program Files\Splunk\bin\Python3.exe" "E:\Program Files\Splunk\etc\apps\Splunk_TA_microsoft-cloudservices\bin\splunk_ta_mscs_rh_azureaccount.py" persistent}: File "E:\Program Files\Splunk\etc\apps\Splunk_TA_microsoft-cloudservices\bin\splunk_ta_mscs_rh_azureaccount.py", line 19, in <module> 02-11-2021 11:12:29.574 -0800 ERROR AdminManagerExternal - Received malformed XML from external handler: 02-11-2021 11:12:29.574 -0800 ERROR AdminManagerExternal - Unable to xml-parse the following data: %s 02-11-2021 11:12:29.574 -0800 ERROR PersistentScript - From {"E:\Program Files\Splunk\bin\Python3.exe" "E:\Program Files\Splunk\etc\apps\Splunk_TA_microsoft-cloudservices\bin\splunk_ta_mscs_rh_azureaccount.py" persistent}: from mscs_util import mscs_consts, get_proxy_info_from_endpoint, get_logger, check_account_isvalid 02-11-2021 11:12:29.574 -0800 ERROR PersistentScript - From {"E:\Program Files\Splunk\bin\Python3.exe" "E:\Program Files\Splunk\etc\apps\Splunk_TA_microsoft-cloudservices\bin\splunk_ta_mscs_rh_azureaccount.py" persistent}: File "E:\Program Files\Splunk\etc\apps\Splunk_TA_microsoft-cloudservices\bin\splunktamscs\mscs_util.py", line 13, in <module> 02-11-2021 11:12:29.574 -0800 ERROR PersistentScript - From {"E:\Program Files\Splunk\bin\Python3.exe" "E:\Program Files\Splunk\etc\apps\Splunk_TA_microsoft-cloudservices\bin\splunk_ta_mscs_rh_azureaccount.py" persistent}: from splunk.appserver.mrsparkle.lib.util import make_splunkhome_path 02-11-2021 11:12:29.574 -0800 ERROR PersistentScript - From {"E:\Program Files\Splunk\bin\Python3.exe" "E:\Program Files\Splunk\etc\apps\Splunk_TA_microsoft-cloudservices\bin\splunk_ta_mscs_rh_azureaccount.py" persistent}: File "E:\Program Files\Splunk\Python-3.7\lib\site-packages\splunk\appserver\mrsparkle\lib\util.py", line 30, in <module> 02-11-2021 11:12:29.574 -0800 ERROR PersistentScript - From {"E:\Program Files\Splunk\bin\Python3.exe" "E:\Program Files\Splunk\etc\apps\Splunk_TA_microsoft-cloudservices\bin\splunk_ta_mscs_rh_azureaccount.py" persistent}: import splunk.search.Parser 02-11-2021 11:12:29.574 -0800 ERROR PersistentScript - From {"E:\Program Files\Splunk\bin\Python3.exe" "E:\Program Files\Splunk\etc\apps\Splunk_TA_microsoft-cloudservices\bin\splunk_ta_mscs_rh_azureaccount.py" persistent}: File "E:\Program Files\Splunk\Python-3.7\lib\site-packages\splunk\search\Parser.py", line 7, in <module> 02-11-2021 11:12:29.574 -0800 ERROR PersistentScript - From {"E:\Program Files\Splunk\bin\Python3.exe" "E:\Program Files\Splunk\etc\apps\Splunk_TA_microsoft-cloudservices\bin\splunk_ta_mscs_rh_azureaccount.py" persistent}: import httplib2 02-11-2021 11:12:29.574 -0800 ERROR PersistentScript - From {"E:\Program Files\Splunk\bin\Python3.exe" "E:\Program Files\Splunk\etc\apps\Splunk_TA_microsoft-cloudservices\bin\splunk_ta_mscs_rh_azureaccount.py" persistent}: File "E:\Program Files\Splunk\etc\apps\Splunk_TA_microsoft-cloudservices\bin\splunktamscs\httplib2\__init__.py", line 28, in <module> 02-11-2021 11:12:29.574 -0800 ERROR PersistentScript - From {"E:\Program Files\Splunk\bin\Python3.exe" "E:\Program Files\Splunk\etc\apps\Splunk_TA_microsoft-cloudservices\bin\splunk_ta_mscs_rh_azureaccount.py" persistent}: import email.FeedParser 02-11-2021 11:12:29.574 -0800 ERROR PersistentScript - From {"E:\Program Files\Splunk\bin\Python3.exe" "E:\Program Files\Splunk\etc\apps\Splunk_TA_microsoft-cloudservices\bin\splunk_ta_mscs_rh_azureaccount.py" persistent}: ModuleNotFoundError: No module named 'email.FeedParser' 02-11-2021 11:12:30.629 -0800 INFO ClientSessionsManager - Adding client: ip=10. uts=windows-x64 id=53CDFDC2-FA78-4D51-B339-B52DFC28F996 name=53CDFDC2-FA78-4D51-B339-B52DFC28F996
... View more