Splunk Enterprise

Updated Splunk Enterprise and Palo Also App crashing

eportenier
Explorer

We went through an upgrade to the latest version of Slunk Enterprise, no problem. However, when we started to upgrade the apps  - specifically a Palo Alto plug-in app - Splunk crashed and now the splunkd service will not stay started. I'm not sure which logs to look at, either, to try to figure out what happened. I have gone through the splunkd.log file but all I see is a bunch of errors (shown below). I know that this isn't much to go on, but can anyone help walk me through trying to troubleshoot this? 

Thank you in advance,

Eric


02-11-2021 11:12:26.015 -0800 ERROR PersistentScript - From {"E:\Program Files\Splunk\bin\Python3.exe" "E:\Program Files\Splunk\etc\apps\Splunk_TA_microsoft-cloudservices\bin\splunk_ta_mscs_rh_azureaccount.py" persistent}: from splunk.appserver.mrsparkle.lib.util import make_splunkhome_path
02-11-2021 11:12:26.015 -0800 ERROR PersistentScript - From {"E:\Program Files\Splunk\bin\Python3.exe" "E:\Program Files\Splunk\etc\apps\Splunk_TA_microsoft-cloudservices\bin\splunk_ta_mscs_rh_azureaccount.py" persistent}: File "E:\Program Files\Splunk\Python-3.7\lib\site-packages\splunk\appserver\mrsparkle\lib\util.py", line 30, in <module>
02-11-2021 11:12:26.015 -0800 ERROR PersistentScript - From {"E:\Program Files\Splunk\bin\Python3.exe" "E:\Program Files\Splunk\etc\apps\Splunk_TA_microsoft-cloudservices\bin\splunk_ta_mscs_rh_azureaccount.py" persistent}: import splunk.search.Parser
02-11-2021 11:12:26.015 -0800 ERROR PersistentScript - From {"E:\Program Files\Splunk\bin\Python3.exe" "E:\Program Files\Splunk\etc\apps\Splunk_TA_microsoft-cloudservices\bin\splunk_ta_mscs_rh_azureaccount.py" persistent}: File "E:\Program Files\Splunk\Python-3.7\lib\site-packages\splunk\search\Parser.py", line 7, in <module>
02-11-2021 11:12:26.015 -0800 ERROR PersistentScript - From {"E:\Program Files\Splunk\bin\Python3.exe" "E:\Program Files\Splunk\etc\apps\Splunk_TA_microsoft-cloudservices\bin\splunk_ta_mscs_rh_azureaccount.py" persistent}: import httplib2
02-11-2021 11:12:26.015 -0800 ERROR PersistentScript - From {"E:\Program Files\Splunk\bin\Python3.exe" "E:\Program Files\Splunk\etc\apps\Splunk_TA_microsoft-cloudservices\bin\splunk_ta_mscs_rh_azureaccount.py" persistent}: File "E:\Program Files\Splunk\etc\apps\Splunk_TA_microsoft-cloudservices\bin\splunktamscs\httplib2\__init__.py", line 28, in <module>
02-11-2021 11:12:26.015 -0800 ERROR PersistentScript - From {"E:\Program Files\Splunk\bin\Python3.exe" "E:\Program Files\Splunk\etc\apps\Splunk_TA_microsoft-cloudservices\bin\splunk_ta_mscs_rh_azureaccount.py" persistent}: import email.FeedParser
02-11-2021 11:12:26.015 -0800 ERROR PersistentScript - From {"E:\Program Files\Splunk\bin\Python3.exe" "E:\Program Files\Splunk\etc\apps\Splunk_TA_microsoft-cloudservices\bin\splunk_ta_mscs_rh_azureaccount.py" persistent}: ModuleNotFoundError: No module named 'email.FeedParser'
02-11-2021 11:12:26.513 -0800 WARN PersistentScript - Process {"E:\Program Files\Splunk\bin\Python3.exe" "E:\Program Files\Splunk\etc\apps\Splunk_TA_microsoft-cloudservices\bin\splunk_ta_mscs_rh_azureaccount.py" persistent}: exited with code 1
02-11-2021 11:12:27.014 -0800 ERROR PersistentScript - From {"E:\Program Files\Splunk\bin\Python3.exe" "E:\Program Files\Splunk\etc\apps\Splunk_TA_microsoft-cloudservices\bin\splunk_ta_mscs_rh_azureaccount.py" persistent}: Traceback (most recent call last):
02-11-2021 11:12:27.014 -0800 ERROR AdminManagerExternal - Received malformed XML from external handler:
02-11-2021 11:12:27.014 -0800 ERROR AdminManagerExternal - Unable to xml-parse the following data: %s
02-11-2021 11:12:27.014 -0800 ERROR PersistentScript - From {"E:\Program Files\Splunk\bin\Python3.exe" "E:\Program Files\Splunk\etc\apps\Splunk_TA_microsoft-cloudservices\bin\splunk_ta_mscs_rh_azureaccount.py" persistent}: File "E:\Program Files\Splunk\etc\apps\Splunk_TA_microsoft-cloudservices\bin\splunk_ta_mscs_rh_azureaccount.py", line 19, in <module>
02-11-2021 11:12:27.014 -0800 ERROR PersistentScript - From {"E:\Program Files\Splunk\bin\Python3.exe" "E:\Program Files\Splunk\etc\apps\Splunk_TA_microsoft-cloudservices\bin\splunk_ta_mscs_rh_azureaccount.py" persistent}: from mscs_util import mscs_consts, get_proxy_info_from_endpoint, get_logger, check_account_isvalid
02-11-2021 11:12:27.014 -0800 ERROR PersistentScript - From {"E:\Program Files\Splunk\bin\Python3.exe" "E:\Program Files\Splunk\etc\apps\Splunk_TA_microsoft-cloudservices\bin\splunk_ta_mscs_rh_azureaccount.py" persistent}: File "E:\Program Files\Splunk\etc\apps\Splunk_TA_microsoft-cloudservices\bin\splunktamscs\mscs_util.py", line 13, in <module>
02-11-2021 11:12:27.014 -0800 ERROR PersistentScript - From {"E:\Program Files\Splunk\bin\Python3.exe" "E:\Program Files\Splunk\etc\apps\Splunk_TA_microsoft-cloudservices\bin\splunk_ta_mscs_rh_azureaccount.py" persistent}: from splunk.appserver.mrsparkle.lib.util import make_splunkhome_path
02-11-2021 11:12:27.014 -0800 ERROR PersistentScript - From {"E:\Program Files\Splunk\bin\Python3.exe" "E:\Program Files\Splunk\etc\apps\Splunk_TA_microsoft-cloudservices\bin\splunk_ta_mscs_rh_azureaccount.py" persistent}: File "E:\Program Files\Splunk\Python-3.7\lib\site-packages\splunk\appserver\mrsparkle\lib\util.py", line 30, in <module>
02-11-2021 11:12:27.014 -0800 ERROR PersistentScript - From {"E:\Program Files\Splunk\bin\Python3.exe" "E:\Program Files\Splunk\etc\apps\Splunk_TA_microsoft-cloudservices\bin\splunk_ta_mscs_rh_azureaccount.py" persistent}: import splunk.search.Parser
02-11-2021 11:12:27.014 -0800 ERROR PersistentScript - From {"E:\Program Files\Splunk\bin\Python3.exe" "E:\Program Files\Splunk\etc\apps\Splunk_TA_microsoft-cloudservices\bin\splunk_ta_mscs_rh_azureaccount.py" persistent}: File "E:\Program Files\Splunk\Python-3.7\lib\site-packages\splunk\search\Parser.py", line 7, in <module>
02-11-2021 11:12:27.014 -0800 ERROR PersistentScript - From {"E:\Program Files\Splunk\bin\Python3.exe" "E:\Program Files\Splunk\etc\apps\Splunk_TA_microsoft-cloudservices\bin\splunk_ta_mscs_rh_azureaccount.py" persistent}: import httplib2
02-11-2021 11:12:27.014 -0800 ERROR PersistentScript - From {"E:\Program Files\Splunk\bin\Python3.exe" "E:\Program Files\Splunk\etc\apps\Splunk_TA_microsoft-cloudservices\bin\splunk_ta_mscs_rh_azureaccount.py" persistent}: File "E:\Program Files\Splunk\etc\apps\Splunk_TA_microsoft-cloudservices\bin\splunktamscs\httplib2\__init__.py", line 28, in <module>
02-11-2021 11:12:27.014 -0800 ERROR PersistentScript - From {"E:\Program Files\Splunk\bin\Python3.exe" "E:\Program Files\Splunk\etc\apps\Splunk_TA_microsoft-cloudservices\bin\splunk_ta_mscs_rh_azureaccount.py" persistent}: import email.FeedParser
02-11-2021 11:12:27.014 -0800 ERROR PersistentScript - From {"E:\Program Files\Splunk\bin\Python3.exe" "E:\Program Files\Splunk\etc\apps\Splunk_TA_microsoft-cloudservices\bin\splunk_ta_mscs_rh_azureaccount.py" persistent}: ModuleNotFoundError: No module named 'email.FeedParser'
02-11-2021 11:12:27.014 -0800 WARN PersistentScript - Process {"E:\Program Files\Splunk\bin\Python3.exe" "E:\Program Files\Splunk\etc\apps\Splunk_TA_microsoft-cloudservices\bin\splunk_ta_mscs_rh_azureaccount.py" persistent}: exited with code 1
02-11-2021 11:12:27.051 -0800 ERROR AdminManagerExternal - Received malformed XML from external handler:
02-11-2021 11:12:27.051 -0800 ERROR AdminManagerExternal - Unable to xml-parse the following data: %s
02-11-2021 11:12:29.073 -0800 WARN PersistentScript - Process {"E:\Program Files\Splunk\bin\Python3.exe" "E:\Program Files\Splunk\etc\apps\Splunk_TA_microsoft-cloudservices\bin\splunk_ta_mscs_rh_azureaccount.py" persistent}: exited with code 1
02-11-2021 11:12:29.574 -0800 ERROR PersistentScript - From {"E:\Program Files\Splunk\bin\Python3.exe" "E:\Program Files\Splunk\etc\apps\Splunk_TA_microsoft-cloudservices\bin\splunk_ta_mscs_rh_azureaccount.py" persistent}: Traceback (most recent call last):
02-11-2021 11:12:29.574 -0800 ERROR PersistentScript - From {"E:\Program Files\Splunk\bin\Python3.exe" "E:\Program Files\Splunk\etc\apps\Splunk_TA_microsoft-cloudservices\bin\splunk_ta_mscs_rh_azureaccount.py" persistent}: File "E:\Program Files\Splunk\etc\apps\Splunk_TA_microsoft-cloudservices\bin\splunk_ta_mscs_rh_azureaccount.py", line 19, in <module>
02-11-2021 11:12:29.574 -0800 ERROR AdminManagerExternal - Received malformed XML from external handler:
02-11-2021 11:12:29.574 -0800 ERROR AdminManagerExternal - Unable to xml-parse the following data: %s
02-11-2021 11:12:29.574 -0800 ERROR PersistentScript - From {"E:\Program Files\Splunk\bin\Python3.exe" "E:\Program Files\Splunk\etc\apps\Splunk_TA_microsoft-cloudservices\bin\splunk_ta_mscs_rh_azureaccount.py" persistent}: from mscs_util import mscs_consts, get_proxy_info_from_endpoint, get_logger, check_account_isvalid
02-11-2021 11:12:29.574 -0800 ERROR PersistentScript - From {"E:\Program Files\Splunk\bin\Python3.exe" "E:\Program Files\Splunk\etc\apps\Splunk_TA_microsoft-cloudservices\bin\splunk_ta_mscs_rh_azureaccount.py" persistent}: File "E:\Program Files\Splunk\etc\apps\Splunk_TA_microsoft-cloudservices\bin\splunktamscs\mscs_util.py", line 13, in <module>
02-11-2021 11:12:29.574 -0800 ERROR PersistentScript - From {"E:\Program Files\Splunk\bin\Python3.exe" "E:\Program Files\Splunk\etc\apps\Splunk_TA_microsoft-cloudservices\bin\splunk_ta_mscs_rh_azureaccount.py" persistent}: from splunk.appserver.mrsparkle.lib.util import make_splunkhome_path
02-11-2021 11:12:29.574 -0800 ERROR PersistentScript - From {"E:\Program Files\Splunk\bin\Python3.exe" "E:\Program Files\Splunk\etc\apps\Splunk_TA_microsoft-cloudservices\bin\splunk_ta_mscs_rh_azureaccount.py" persistent}: File "E:\Program Files\Splunk\Python-3.7\lib\site-packages\splunk\appserver\mrsparkle\lib\util.py", line 30, in <module>
02-11-2021 11:12:29.574 -0800 ERROR PersistentScript - From {"E:\Program Files\Splunk\bin\Python3.exe" "E:\Program Files\Splunk\etc\apps\Splunk_TA_microsoft-cloudservices\bin\splunk_ta_mscs_rh_azureaccount.py" persistent}: import splunk.search.Parser
02-11-2021 11:12:29.574 -0800 ERROR PersistentScript - From {"E:\Program Files\Splunk\bin\Python3.exe" "E:\Program Files\Splunk\etc\apps\Splunk_TA_microsoft-cloudservices\bin\splunk_ta_mscs_rh_azureaccount.py" persistent}: File "E:\Program Files\Splunk\Python-3.7\lib\site-packages\splunk\search\Parser.py", line 7, in <module>
02-11-2021 11:12:29.574 -0800 ERROR PersistentScript - From {"E:\Program Files\Splunk\bin\Python3.exe" "E:\Program Files\Splunk\etc\apps\Splunk_TA_microsoft-cloudservices\bin\splunk_ta_mscs_rh_azureaccount.py" persistent}: import httplib2
02-11-2021 11:12:29.574 -0800 ERROR PersistentScript - From {"E:\Program Files\Splunk\bin\Python3.exe" "E:\Program Files\Splunk\etc\apps\Splunk_TA_microsoft-cloudservices\bin\splunk_ta_mscs_rh_azureaccount.py" persistent}: File "E:\Program Files\Splunk\etc\apps\Splunk_TA_microsoft-cloudservices\bin\splunktamscs\httplib2\__init__.py", line 28, in <module>
02-11-2021 11:12:29.574 -0800 ERROR PersistentScript - From {"E:\Program Files\Splunk\bin\Python3.exe" "E:\Program Files\Splunk\etc\apps\Splunk_TA_microsoft-cloudservices\bin\splunk_ta_mscs_rh_azureaccount.py" persistent}: import email.FeedParser
02-11-2021 11:12:29.574 -0800 ERROR PersistentScript - From {"E:\Program Files\Splunk\bin\Python3.exe" "E:\Program Files\Splunk\etc\apps\Splunk_TA_microsoft-cloudservices\bin\splunk_ta_mscs_rh_azureaccount.py" persistent}: ModuleNotFoundError: No module named 'email.FeedParser'
02-11-2021 11:12:30.629 -0800 INFO ClientSessionsManager - Adding client: ip=10. uts=windows-x64 id=53CDFDC2-FA78-4D51-B339-B52DFC28F996 name=53CDFDC2-FA78-4D51-B339-B52DFC28F996

Labels (1)
0 Karma
1 Solution

scelikok
SplunkTrust
SplunkTrust

Hi @eportenier,

It seems there is an Python3 incompatibility on Splunk_TA_microsoft-cloudservices app. SplunkWeb cannot start. Can you please try disabling this app and restart?

To prevent issues starting Splunk Web, revise apps for Python 3 compatibility. If an app cannot be upgraded, it must be removed for Splunk Web to start.

https://docs.splunk.com/Documentation/Splunk/8.1.2/Python3Migration/ChangesEnterprise#Splunk_Web 

If this reply helps you an upvote and "Accept as Solution" is appreciated.

View solution in original post

scelikok
SplunkTrust
SplunkTrust

Hi @eportenier,

It seems there is an Python3 incompatibility on Splunk_TA_microsoft-cloudservices app. SplunkWeb cannot start. Can you please try disabling this app and restart?

To prevent issues starting Splunk Web, revise apps for Python 3 compatibility. If an app cannot be upgraded, it must be removed for Splunk Web to start.

https://docs.splunk.com/Documentation/Splunk/8.1.2/Python3Migration/ChangesEnterprise#Splunk_Web 

If this reply helps you an upvote and "Accept as Solution" is appreciated.

eportenier
Explorer

That's what it was. 🙂 I was able to find the App folder that had the installed apps in it. I removed the "bad" app and started the service... it came back! Thank you very much for your assistance!!

Eric

0 Karma

isoutamo
SplunkTrust
SplunkTrust

Hi

before update to 8.x.x you should check readiness of apps with https://splunkbase.splunk.com/app/4698/

r. Ismo

0 Karma

eportenier
Explorer

Thank you for your reply! I hate sounding stupid, but where do I disable an app from starting? I'm assuming that there is a configuration file somewhere? Sorry for the dumb question... I know I should probably know this but I haven't really had to delve into the inner workings of Splunk lately. 

 

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...