Palo Alto Networks Add-on error with rest_migration.py

[matthewroberson]

I updated my Palo Alto Networks Add-on to version 6.3.1 and now I'm seeing the errors below in splunkd.log on the search head cluster members the add-on is deployed to.

09-03-2020 09:54:10.323 -0500 ERROR AdminManagerExternal - Stack trace from python handler:\nTraceback (most recent call last):\n  File "/opt/splunk/etc/apps/Splunk_TA_paloalto/bin/splunk_ta_paloalto/aob_py3/splunk_aoblib/rest_migration.py", line 19, in handle\n    return func(*args, **kwargs)\n  File "/opt/splunk/etc/apps/Splunk_TA_paloalto/bin/splunk_ta_paloalto/aob_py3/splunk_aoblib/rest_migration.py", line 71, in _migrate\n    self._migrate_conf_credential()\n  File "/opt/splunk/etc/apps/Splunk_TA_paloalto/bin/splunk_ta_paloalto/aob_py3/splunk_aoblib/rest_migration.py", line 161, in _migrate_conf_credential\n    conf_file, stanzas = self._load_conf(conf_file_name)\n  File "/opt/splunk/etc/apps/Splunk_TA_paloalto/bin/splunk_ta_paloalto/aob_py3/splunk_aoblib/rest_migration.py", line 178, in _load_conf\n    stanzas = conf_file.get_all()\n  File "/opt/splunk/etc/apps/Splunk_TA_paloalto/bin/splunk_ta_paloalto/aob_py3/solnlib/utils.py", line 159, in wrapper\n    return func(*args, **kwargs)\n  File "/opt/splunk/etc/apps/Splunk_TA_paloalto/bin/splunk_ta_paloalto/aob_py3/solnlib/conf_manager.py", line 241, in get_all\n    key_values = self._decrypt_stanza(name, stanza_mgr.content)\n  File "/opt/splunk/etc/apps/Splunk_TA_paloalto/bin/splunk_ta_paloalto/aob_py3/solnlib/conf_manager.py", line 126, in _decrypt_stanza\n    self._cred_mgr.get_password(stanza_name))\n  File "/opt/splunk/lib/python3.7/json/__init__.py", line 348, in loads\n    return _default_decoder.decode(s)\n  File "/opt/splunk/lib/python3.7/json/decoder.py", line 337, in decode\n    obj, end = self.raw_decode(s, idx=_w(s, 0).end())\n  File "/opt/splunk/lib/python3.7/json/decoder.py", line 355, in raw_decode\n    raise JSONDecodeError("Expecting value", s, err.value) from None\njson.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n  File "/opt/splunk/lib/python3.7/site-packages/splunk/admin.py", line 148, in init\n    hand.execute(info)\n  File "/opt/splunk/lib/python3.7/site-packages/splunk/admin.py", line 634, in execute\n    if self.requestedAction == ACTION_LIST:     self.handleList(confInfo)\n  File "/opt/splunk/etc/apps/Splunk_TA_paloalto/bin/splunk_ta_paloalto/aob_py3/splunk_aoblib/rest_migration.py", line 36, in handleList\n    self._migrate()\n  File "/opt/splunk/etc/apps/Splunk_TA_paloalto/bin/splunk_ta_paloalto/aob_py3/splunk_aoblib/rest_migration.py", line 23, in handle\n    'Migrating failed. %s' % traceback.format_exc()\nsplunktaucclib.rest_handler.error.RestError: REST Error [500]: Internal Server Error -- Migrating failed. Traceback (most recent call last):\n  File "/opt/splunk/etc/apps/Splunk_TA_paloalto/bin/splunk_ta_paloalto/aob_py3/splunk_aoblib/rest_migration.py", line 19, in handle\n    return func(*args, **kwargs)\n  File "/opt/splunk/etc/apps/Splunk_TA_paloalto/bin/splunk_ta_paloalto/aob_py3/splunk_aoblib/rest_migration.py", line 71, in _migrate\n    self._migrate_conf_credential()\n  File "/opt/splunk/etc/apps/Splunk_TA_paloalto/bin/splunk_ta_paloalto/aob_py3/splunk_aoblib/rest_migration.py", line 161, in _migrate_conf_credential\n    conf_file, stanzas = self._load_conf(conf_file_name)\n  File "/opt/splunk/etc/apps/Splunk_TA_paloalto/bin/splunk_ta_paloalto/aob_py3/splunk_aoblib/rest_migration.py", line 178, in _load_conf\n    stanzas = conf_file.get_all()\n  File "/opt/splunk/etc/apps/Splunk_TA_paloalto/bin/splunk_ta_paloalto/aob_py3/solnlib/utils.py", line 159, in wrapper\n    return func(*args, **kwargs)\n  File "/opt/splunk/etc/apps/Splunk_TA_paloalto/bin/splunk_ta_paloalto/aob_py3/solnlib/conf_manager.py", line 241, in get_all\n    key_values = self._decrypt_stanza(name, stanza_mgr.content)\n  File "/opt/splunk/etc/apps/Splunk_TA_paloalto/bin/splunk_ta_paloalto/aob_py3/solnlib/conf_manager.py", line 126, in _decrypt_stanza\n    self._cred_mgr.get_password(stanza_name))\n  File "/opt/splunk/lib/python3.7/json/__init__.py", line 348, in loads\n    return _default_decoder.decode(s)\n  File "/opt/splunk/lib/python3.7/json/decoder.py", line 337, in decode\n    obj, end = self.raw_decode(s, idx=_w(s, 0).end())\n  File "/opt/splunk/lib/python3.7/json/decoder.py", line 355, in raw_decode\n    raise JSONDecodeError("Expecting value", s, err.value) from None\njson.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)\n\n
09-03-2020 09:54:10.323 -0500 ERROR AdminManagerExternal - Unexpected error "<class 'splunktaucclib.rest_handler.error.RestError'>" from python handler: "REST Error [500]: Internal Server Error -- Migrating failed. Traceback (most recent call last):\n  File "/opt/splunk/etc/apps/Splunk_TA_paloalto/bin/splunk_ta_paloalto/aob_py3/splunk_aoblib/rest_migration.py", line 19, in handle\n    return func(*args, **kwargs)\n  File "/opt/splunk/etc/apps/Splunk_TA_paloalto/bin/splunk_ta_paloalto/aob_py3/splunk_aoblib/rest_migration.py", line 71, in _migrate\n    self._migrate_conf_credential()\n  File "/opt/splunk/etc/apps/Splunk_TA_paloalto/bin/splunk_ta_paloalto/aob_py3/splunk_aoblib/rest_migration.py", line 161, in _migrate_conf_credential\n    conf_file, stanzas = self._load_conf(conf_file_name)\n  File "/opt/splunk/etc/apps/Splunk_TA_paloalto/bin/splunk_ta_paloalto/aob_py3/splunk_aoblib/rest_migration.py", line 178, in _load_conf\n    stanzas = conf_file.get_all()\n  File "/opt/splunk/etc/apps/Splunk_TA_paloalto/bin/splunk_ta_paloalto/aob_py3/solnlib/utils.py", line 159, in wrapper\n    return func(*args, **kwargs)\n  File "/opt/splunk/etc/apps/Splunk_TA_paloalto/bin/splunk_ta_paloalto/aob_py3/solnlib/conf_manager.py", line 241, in get_all\n    key_values = self._decrypt_stanza(name, stanza_mgr.content)\n  File "/opt/splunk/etc/apps/Splunk_TA_paloalto/bin/splunk_ta_paloalto/aob_py3/solnlib/conf_manager.py", line 126, in _decrypt_stanza\n    self._cred_mgr.get_password(stanza_name))\n  File "/opt/splunk/lib/python3.7/json/__init__.py", line 348, in loads\n    return _default_decoder.decode(s)\n  File "/opt/splunk/lib/python3.7/json/decoder.py", line 337, in decode\n    obj, end = self.raw_decode(s, idx=_w(s, 0).end())\n  File "/opt/splunk/lib/python3.7/json/decoder.py", line 355, in raw_decode\n    raise JSONDecodeError("Expecting value", s, err.value) from None\njson.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)\n".  See splunkd.log for more details.

I took a look at rest_migration.py and it looks to me like it's looking for credentials from an older version of the TA that wasn't installed on my search heads (I'm not great with python, so I could be wrong). The add-on is deployed to a 4 member search head cluster with my deployer. Anyone have any ideas on how to resolve this? As it is, when i try to configure accounts or add-on settings in the app i just get a spinning wheel that says loading.

[matthewroberson · Accepted solution]

I resolved the issue by removing app and add-on from /opt/splunk/etc/shcluster/apps on the deployer and deploying ( to remove them from the search head ). Then I put the app and add-on back in /opt/splunk/etc/shcluster/apps and redeployed. This resolved the issue.

[thambisetty]

What credentials are you talking about?

are you making use of alert actions to manage your firewall from splunk TA?

[matthewroberson]

    def get_legacy_passwords(self):
        if self.legacy_passwords is None:
            self.legacy_passwords = {}
            for pwd in self.client.storage_passwords.list(count=-1):
                if pwd.realm == self.base_app_name:
                    self.legacy_passwords[pwd.username] = pwd
        return self.legacy_passwords

The above is what I noticed and also the snippet from the error messages below:

self._migrate_conf_credential()\n  File "/opt/splunk/etc/apps/Splunk_TA_paloalto/bin/splunk_ta_paloalto/aob_py3/splunk_aoblib/rest_migration.py

I don't have the best grasp of Python, so maybe I'm way off base?

[matthewroberson]

I am not making use of alert actions to manage your firewall from splunk TA. Intended to include that in my original reply...

[thambisetty]

what is your splunk enterprise version ?

https://community.splunk.com/t5/All-Apps-and-Add-ons/Palo-Alto-Networks-Add-on-6-3-1-upgrade-on-Splu...

[matthewroberson]

8.0.5 running on linux servers...