I updated my Palo Alto Networks Add-on to version 6.3.1 and now I'm seeing the errors below in splunkd.log on the search head cluster members the add-on is deployed to.
09-03-2020 09:54:10.323 -0500 ERROR AdminManagerExternal - Stack trace from python handler:\nTraceback (most recent call last):\n File "/opt/splunk/etc/apps/Splunk_TA_paloalto/bin/splunk_ta_paloalto/aob_py3/splunk_aoblib/rest_migration.py", line 19, in handle\n return func(*args, **kwargs)\n File "/opt/splunk/etc/apps/Splunk_TA_paloalto/bin/splunk_ta_paloalto/aob_py3/splunk_aoblib/rest_migration.py", line 71, in _migrate\n self._migrate_conf_credential()\n File "/opt/splunk/etc/apps/Splunk_TA_paloalto/bin/splunk_ta_paloalto/aob_py3/splunk_aoblib/rest_migration.py", line 161, in _migrate_conf_credential\n conf_file, stanzas = self._load_conf(conf_file_name)\n File "/opt/splunk/etc/apps/Splunk_TA_paloalto/bin/splunk_ta_paloalto/aob_py3/splunk_aoblib/rest_migration.py", line 178, in _load_conf\n stanzas = conf_file.get_all()\n File "/opt/splunk/etc/apps/Splunk_TA_paloalto/bin/splunk_ta_paloalto/aob_py3/solnlib/utils.py", line 159, in wrapper\n return func(*args, **kwargs)\n File "/opt/splunk/etc/apps/Splunk_TA_paloalto/bin/splunk_ta_paloalto/aob_py3/solnlib/conf_manager.py", line 241, in get_all\n key_values = self._decrypt_stanza(name, stanza_mgr.content)\n File "/opt/splunk/etc/apps/Splunk_TA_paloalto/bin/splunk_ta_paloalto/aob_py3/solnlib/conf_manager.py", line 126, in _decrypt_stanza\n self._cred_mgr.get_password(stanza_name))\n File "/opt/splunk/lib/python3.7/json/__init__.py", line 348, in loads\n return _default_decoder.decode(s)\n File "/opt/splunk/lib/python3.7/json/decoder.py", line 337, in decode\n obj, end = self.raw_decode(s, idx=_w(s, 0).end())\n File "/opt/splunk/lib/python3.7/json/decoder.py", line 355, in raw_decode\n raise JSONDecodeError("Expecting value", s, err.value) from None\njson.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File "/opt/splunk/lib/python3.7/site-packages/splunk/admin.py", line 148, in init\n hand.execute(info)\n File "/opt/splunk/lib/python3.7/site-packages/splunk/admin.py", line 634, in execute\n if self.requestedAction == ACTION_LIST: self.handleList(confInfo)\n File "/opt/splunk/etc/apps/Splunk_TA_paloalto/bin/splunk_ta_paloalto/aob_py3/splunk_aoblib/rest_migration.py", line 36, in handleList\n self._migrate()\n File "/opt/splunk/etc/apps/Splunk_TA_paloalto/bin/splunk_ta_paloalto/aob_py3/splunk_aoblib/rest_migration.py", line 23, in handle\n 'Migrating failed. %s' % traceback.format_exc()\nsplunktaucclib.rest_handler.error.RestError: REST Error [500]: Internal Server Error -- Migrating failed. Traceback (most recent call last):\n File "/opt/splunk/etc/apps/Splunk_TA_paloalto/bin/splunk_ta_paloalto/aob_py3/splunk_aoblib/rest_migration.py", line 19, in handle\n return func(*args, **kwargs)\n File "/opt/splunk/etc/apps/Splunk_TA_paloalto/bin/splunk_ta_paloalto/aob_py3/splunk_aoblib/rest_migration.py", line 71, in _migrate\n self._migrate_conf_credential()\n File "/opt/splunk/etc/apps/Splunk_TA_paloalto/bin/splunk_ta_paloalto/aob_py3/splunk_aoblib/rest_migration.py", line 161, in _migrate_conf_credential\n conf_file, stanzas = self._load_conf(conf_file_name)\n File "/opt/splunk/etc/apps/Splunk_TA_paloalto/bin/splunk_ta_paloalto/aob_py3/splunk_aoblib/rest_migration.py", line 178, in _load_conf\n stanzas = conf_file.get_all()\n File "/opt/splunk/etc/apps/Splunk_TA_paloalto/bin/splunk_ta_paloalto/aob_py3/solnlib/utils.py", line 159, in wrapper\n return func(*args, **kwargs)\n File "/opt/splunk/etc/apps/Splunk_TA_paloalto/bin/splunk_ta_paloalto/aob_py3/solnlib/conf_manager.py", line 241, in get_all\n key_values = self._decrypt_stanza(name, stanza_mgr.content)\n File "/opt/splunk/etc/apps/Splunk_TA_paloalto/bin/splunk_ta_paloalto/aob_py3/solnlib/conf_manager.py", line 126, in _decrypt_stanza\n self._cred_mgr.get_password(stanza_name))\n File "/opt/splunk/lib/python3.7/json/__init__.py", line 348, in loads\n return _default_decoder.decode(s)\n File "/opt/splunk/lib/python3.7/json/decoder.py", line 337, in decode\n obj, end = self.raw_decode(s, idx=_w(s, 0).end())\n File "/opt/splunk/lib/python3.7/json/decoder.py", line 355, in raw_decode\n raise JSONDecodeError("Expecting value", s, err.value) from None\njson.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)\n\n
09-03-2020 09:54:10.323 -0500 ERROR AdminManagerExternal - Unexpected error "<class 'splunktaucclib.rest_handler.error.RestError'>" from python handler: "REST Error [500]: Internal Server Error -- Migrating failed. Traceback (most recent call last):\n File "/opt/splunk/etc/apps/Splunk_TA_paloalto/bin/splunk_ta_paloalto/aob_py3/splunk_aoblib/rest_migration.py", line 19, in handle\n return func(*args, **kwargs)\n File "/opt/splunk/etc/apps/Splunk_TA_paloalto/bin/splunk_ta_paloalto/aob_py3/splunk_aoblib/rest_migration.py", line 71, in _migrate\n self._migrate_conf_credential()\n File "/opt/splunk/etc/apps/Splunk_TA_paloalto/bin/splunk_ta_paloalto/aob_py3/splunk_aoblib/rest_migration.py", line 161, in _migrate_conf_credential\n conf_file, stanzas = self._load_conf(conf_file_name)\n File "/opt/splunk/etc/apps/Splunk_TA_paloalto/bin/splunk_ta_paloalto/aob_py3/splunk_aoblib/rest_migration.py", line 178, in _load_conf\n stanzas = conf_file.get_all()\n File "/opt/splunk/etc/apps/Splunk_TA_paloalto/bin/splunk_ta_paloalto/aob_py3/solnlib/utils.py", line 159, in wrapper\n return func(*args, **kwargs)\n File "/opt/splunk/etc/apps/Splunk_TA_paloalto/bin/splunk_ta_paloalto/aob_py3/solnlib/conf_manager.py", line 241, in get_all\n key_values = self._decrypt_stanza(name, stanza_mgr.content)\n File "/opt/splunk/etc/apps/Splunk_TA_paloalto/bin/splunk_ta_paloalto/aob_py3/solnlib/conf_manager.py", line 126, in _decrypt_stanza\n self._cred_mgr.get_password(stanza_name))\n File "/opt/splunk/lib/python3.7/json/__init__.py", line 348, in loads\n return _default_decoder.decode(s)\n File "/opt/splunk/lib/python3.7/json/decoder.py", line 337, in decode\n obj, end = self.raw_decode(s, idx=_w(s, 0).end())\n File "/opt/splunk/lib/python3.7/json/decoder.py", line 355, in raw_decode\n raise JSONDecodeError("Expecting value", s, err.value) from None\njson.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)\n". See splunkd.log for more details.
I took a look at rest_migration.py and it looks to me like it's looking for credentials from an older version of the TA that wasn't installed on my search heads (I'm not great with python, so I could be wrong). The add-on is deployed to a 4 member search head cluster with my deployer. Anyone have any ideas on how to resolve this? As it is, when i try to configure accounts or add-on settings in the app i just get a spinning wheel that says loading.
I resolved the issue by removing app and add-on from /opt/splunk/etc/shcluster/apps on the deployer and deploying ( to remove them from the search head ). Then I put the app and add-on back in /opt/splunk/etc/shcluster/apps and redeployed. This resolved the issue.
I resolved the issue by removing app and add-on from /opt/splunk/etc/shcluster/apps on the deployer and deploying ( to remove them from the search head ). Then I put the app and add-on back in /opt/splunk/etc/shcluster/apps and redeployed. This resolved the issue.
What credentials are you talking about?
are you making use of alert actions to manage your firewall from splunk TA?
def get_legacy_passwords(self):
if self.legacy_passwords is None:
self.legacy_passwords = {}
for pwd in self.client.storage_passwords.list(count=-1):
if pwd.realm == self.base_app_name:
self.legacy_passwords[pwd.username] = pwd
return self.legacy_passwords
The above is what I noticed and also the snippet from the error messages below:
self._migrate_conf_credential()\n File "/opt/splunk/etc/apps/Splunk_TA_paloalto/bin/splunk_ta_paloalto/aob_py3/splunk_aoblib/rest_migration.py
I don't have the best grasp of Python, so maybe I'm way off base?
I am not making use of alert actions to manage your firewall from splunk TA. Intended to include that in my original reply...
what is your splunk enterprise version ?
8.0.5 running on linux servers...