I updated my Palo Alto Networks Add-on to version 6.3.1 and now I'm seeing the errors below in splunkd.log on the search head cluster members the add-on is deployed to. 09-03-2020 09:54:10.323 -0500 ERROR AdminManagerExternal - Stack trace from python handler:\nTraceback (most recent call last):\n File "/opt/splunk/etc/apps/Splunk_TA_paloalto/bin/splunk_ta_paloalto/aob_py3/splunk_aoblib/rest_migration.py", line 19, in handle\n return func(*args, **kwargs)\n File "/opt/splunk/etc/apps/Splunk_TA_paloalto/bin/splunk_ta_paloalto/aob_py3/splunk_aoblib/rest_migration.py", line 71, in _migrate\n self._migrate_conf_credential()\n File "/opt/splunk/etc/apps/Splunk_TA_paloalto/bin/splunk_ta_paloalto/aob_py3/splunk_aoblib/rest_migration.py", line 161, in _migrate_conf_credential\n conf_file, stanzas = self._load_conf(conf_file_name)\n File "/opt/splunk/etc/apps/Splunk_TA_paloalto/bin/splunk_ta_paloalto/aob_py3/splunk_aoblib/rest_migration.py", line 178, in _load_conf\n stanzas = conf_file.get_all()\n File "/opt/splunk/etc/apps/Splunk_TA_paloalto/bin/splunk_ta_paloalto/aob_py3/solnlib/utils.py", line 159, in wrapper\n return func(*args, **kwargs)\n File "/opt/splunk/etc/apps/Splunk_TA_paloalto/bin/splunk_ta_paloalto/aob_py3/solnlib/conf_manager.py", line 241, in get_all\n key_values = self._decrypt_stanza(name, stanza_mgr.content)\n File "/opt/splunk/etc/apps/Splunk_TA_paloalto/bin/splunk_ta_paloalto/aob_py3/solnlib/conf_manager.py", line 126, in _decrypt_stanza\n self._cred_mgr.get_password(stanza_name))\n File "/opt/splunk/lib/python3.7/json/__init__.py", line 348, in loads\n return _default_decoder.decode(s)\n File "/opt/splunk/lib/python3.7/json/decoder.py", line 337, in decode\n obj, end = self.raw_decode(s, idx=_w(s, 0).end())\n File "/opt/splunk/lib/python3.7/json/decoder.py", line 355, in raw_decode\n raise JSONDecodeError("Expecting value", s, err.value) from None\njson.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File "/opt/splunk/lib/python3.7/site-packages/splunk/admin.py", line 148, in init\n hand.execute(info)\n File "/opt/splunk/lib/python3.7/site-packages/splunk/admin.py", line 634, in execute\n if self.requestedAction == ACTION_LIST: self.handleList(confInfo)\n File "/opt/splunk/etc/apps/Splunk_TA_paloalto/bin/splunk_ta_paloalto/aob_py3/splunk_aoblib/rest_migration.py", line 36, in handleList\n self._migrate()\n File "/opt/splunk/etc/apps/Splunk_TA_paloalto/bin/splunk_ta_paloalto/aob_py3/splunk_aoblib/rest_migration.py", line 23, in handle\n 'Migrating failed. %s' % traceback.format_exc()\nsplunktaucclib.rest_handler.error.RestError: REST Error [500]: Internal Server Error -- Migrating failed. Traceback (most recent call last):\n File "/opt/splunk/etc/apps/Splunk_TA_paloalto/bin/splunk_ta_paloalto/aob_py3/splunk_aoblib/rest_migration.py", line 19, in handle\n return func(*args, **kwargs)\n File "/opt/splunk/etc/apps/Splunk_TA_paloalto/bin/splunk_ta_paloalto/aob_py3/splunk_aoblib/rest_migration.py", line 71, in _migrate\n self._migrate_conf_credential()\n File "/opt/splunk/etc/apps/Splunk_TA_paloalto/bin/splunk_ta_paloalto/aob_py3/splunk_aoblib/rest_migration.py", line 161, in _migrate_conf_credential\n conf_file, stanzas = self._load_conf(conf_file_name)\n File "/opt/splunk/etc/apps/Splunk_TA_paloalto/bin/splunk_ta_paloalto/aob_py3/splunk_aoblib/rest_migration.py", line 178, in _load_conf\n stanzas = conf_file.get_all()\n File "/opt/splunk/etc/apps/Splunk_TA_paloalto/bin/splunk_ta_paloalto/aob_py3/solnlib/utils.py", line 159, in wrapper\n return func(*args, **kwargs)\n File "/opt/splunk/etc/apps/Splunk_TA_paloalto/bin/splunk_ta_paloalto/aob_py3/solnlib/conf_manager.py", line 241, in get_all\n key_values = self._decrypt_stanza(name, stanza_mgr.content)\n File "/opt/splunk/etc/apps/Splunk_TA_paloalto/bin/splunk_ta_paloalto/aob_py3/solnlib/conf_manager.py", line 126, in _decrypt_stanza\n self._cred_mgr.get_password(stanza_name))\n File "/opt/splunk/lib/python3.7/json/__init__.py", line 348, in loads\n return _default_decoder.decode(s)\n File "/opt/splunk/lib/python3.7/json/decoder.py", line 337, in decode\n obj, end = self.raw_decode(s, idx=_w(s, 0).end())\n File "/opt/splunk/lib/python3.7/json/decoder.py", line 355, in raw_decode\n raise JSONDecodeError("Expecting value", s, err.value) from None\njson.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)\n\n
09-03-2020 09:54:10.323 -0500 ERROR AdminManagerExternal - Unexpected error "<class 'splunktaucclib.rest_handler.error.RestError'>" from python handler: "REST Error [500]: Internal Server Error -- Migrating failed. Traceback (most recent call last):\n File "/opt/splunk/etc/apps/Splunk_TA_paloalto/bin/splunk_ta_paloalto/aob_py3/splunk_aoblib/rest_migration.py", line 19, in handle\n return func(*args, **kwargs)\n File "/opt/splunk/etc/apps/Splunk_TA_paloalto/bin/splunk_ta_paloalto/aob_py3/splunk_aoblib/rest_migration.py", line 71, in _migrate\n self._migrate_conf_credential()\n File "/opt/splunk/etc/apps/Splunk_TA_paloalto/bin/splunk_ta_paloalto/aob_py3/splunk_aoblib/rest_migration.py", line 161, in _migrate_conf_credential\n conf_file, stanzas = self._load_conf(conf_file_name)\n File "/opt/splunk/etc/apps/Splunk_TA_paloalto/bin/splunk_ta_paloalto/aob_py3/splunk_aoblib/rest_migration.py", line 178, in _load_conf\n stanzas = conf_file.get_all()\n File "/opt/splunk/etc/apps/Splunk_TA_paloalto/bin/splunk_ta_paloalto/aob_py3/solnlib/utils.py", line 159, in wrapper\n return func(*args, **kwargs)\n File "/opt/splunk/etc/apps/Splunk_TA_paloalto/bin/splunk_ta_paloalto/aob_py3/solnlib/conf_manager.py", line 241, in get_all\n key_values = self._decrypt_stanza(name, stanza_mgr.content)\n File "/opt/splunk/etc/apps/Splunk_TA_paloalto/bin/splunk_ta_paloalto/aob_py3/solnlib/conf_manager.py", line 126, in _decrypt_stanza\n self._cred_mgr.get_password(stanza_name))\n File "/opt/splunk/lib/python3.7/json/__init__.py", line 348, in loads\n return _default_decoder.decode(s)\n File "/opt/splunk/lib/python3.7/json/decoder.py", line 337, in decode\n obj, end = self.raw_decode(s, idx=_w(s, 0).end())\n File "/opt/splunk/lib/python3.7/json/decoder.py", line 355, in raw_decode\n raise JSONDecodeError("Expecting value", s, err.value) from None\njson.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)\n". See splunkd.log for more details. I took a look at rest_migration.py and it looks to me like it's looking for credentials from an older version of the TA that wasn't installed on my search heads (I'm not great with python, so I could be wrong). The add-on is deployed to a 4 member search head cluster with my deployer. Anyone have any ideas on how to resolve this? As it is, when i try to configure accounts or add-on settings in the app i just get a spinning wheel that says loading.
... View more