Issues with Splunk Workday Add-on, not saving configuration, spinning wheel 'Loading'

[glowry]

We had the workday app (4.1) running on a HF (6.6.5) that apparently stopped working on Oct 24th.

Error:
File "D:\Program Files\Splunk\etc\apps\TA-workday\bin\ta_workday\solnlib\packages\splunklib\binding.py", line 1221, in request
raise HTTPError(response)
HTTPError: HTTP 503 Service Unavailable -- KV Store initialization failed. Please contact your system administrator.

I am now attempting to move that add-on to a new HF...as recommended by support for another case.

1. I have copied the TA from the HF that was working, restarted Splunk - it didnt work.
2. I have re-installed v 4.1 on the HF from the .tgz file, restarted - it didnt work
3. I have re-installed with v 5.0 on the HF from .tgz, restarted - it didnt work
4. I've pushed the security settings down from the \etc\apps folder down to all subfolders...to eliminate any permissions issues.

When i log into the web ui, go to workday app, then go to configuration> add-on settings...the loading icon appears and shows about 4-5 bars and is stuck...it eventually shows the setting options, i configure them, hit save, but it never saves them.

The ta never attempts to call out to the internet to grab files, looking in the workday ta log, the splunkd and splunkaccess logs i see 404 and 500 http server errors.

splunkd.log:
11-07-2018 01:18:53.847 -0600 ERROR AdminManagerExternal - Unexpected error "" from python handler: "REST Error [500]: Internal Server Error -- Traceback (most recent call last):\n File "D:\Program Files\Splunk\etc\apps\TA-workday\bin\ta_workday\splunktaucclib\rest_handler\handler.py", line 113, in wrapper\n for name, data, acl in meth(self, *args, **kwargs):\n File "D:\Program Files\Splunk\etc\apps\TA-workday\bin\ta_workday\splunktaucclib\rest_handler\handler.py", line 299, in _format_response\n masked = self.rest_credentials.decrypt_for_get(name, data)\n File "D:\Program Files\Splunk\etc\apps\TA-workday\bin\ta_workday\splunktaucclib\rest_handler\credentials.py", line 184, in decrypt_for_get\n clear_password = self._get(name)\n File "D:\Program Files\Splunk\etc\apps\TA-workday\bin\ta_workday\splunktaucclib\rest_handler\credentials.py", line 389, in _get\n string = mgr.get_password(user=context.username())\n File "D:\Program Files\Splunk\etc\apps\TA-workday\bin\ta_workday\solnlib\utils.py", line 154, in wrapper\n return func(*args, **kwargs)\n File "D:\Program Files\Splunk\etc\apps\TA-workday\bin\ta_workday\solnlib\credentials.py", line 118, in get_password\n all_passwords = self._get_all_passwords()\n File "D:\Program Files\Splunk\etc\apps\TA-workday\bin\ta_workday\solnlib\utils.py", line 154, in wrapper\n return func(*args, **kwargs)\n File "D:\Program Files\Splunk\etc\apps\TA-workday\bin\ta_workday\solnlib\credentials.py", line 272, in _get_all_passwords\n clear_password += field_clear[index]\nTypeError: cannot concatenate 'str' and 'NoneType' objects\n". See splunkd.log for more details.

11-07-2018 01:19:05.427 -0600 ERROR AdminManagerExternal - Unexpected error "" from python handler: "REST Error [500]: Internal Server Error -- Traceback (most recent call last):\n File "D:\Program Files\Splunk\etc\apps\TA-workday\bin\ta_workday\splunktaucclib\rest_handler\handler.py", line 113, in wrapper\n for name, data, acl in meth(self, *args, **kwargs):\n File "D:\Program Files\Splunk\etc\apps\TA-workday\bin\ta_workday\splunktaucclib\rest_handler\handler.py", line 299, in _format_response\n masked = self.rest_credentials.decrypt_for_get(name, data)\n File "D:\Program Files\Splunk\etc\apps\TA-workday\bin\ta_workday\splunktaucclib\rest_handler\credentials.py", line 184, in decrypt_for_get\n clear_password = self._get(name)\n File "D:\Program Files\Splunk\etc\apps\TA-workday\bin\ta_workday\splunktaucclib\rest_handler\credentials.py", line 389, in _get\n string = mgr.get_password(user=context.username())\n File "D:\Program Files\Splunk\etc\apps\TA-workday\bin\ta_workday\solnlib\utils.py", line 154, in wrapper\n return func(*args, **kwargs)\n File "D:\Program Files\Splunk\etc\apps\TA-workday\bin\ta_workday\solnlib\credentials.py", line 118, in get_password\n all_passwords = self._get_all_passwords()\n File "D:\Program Files\Splunk\etc\apps\TA-workday\bin\ta_workday\solnlib\utils.py", line 154, in wrapper\n return func(*args, **kwargs)\n File "D:\Program Files\Splunk\etc\apps\TA-workday\bin\ta_workday\solnlib\credentials.py", line 272, in _get_all_passwords\n clear_password += field_clear[index]\nTypeError: cannot concatenate 'str' and 'NoneType' objects\n". See splunkd.log for more details.

11-07-2018 01:23:52.873 -0600 ERROR AdminManagerExternal - Unexpected error "" from python handler: "REST Error [500]: Internal Server Error -- Traceback (most recent call last):\n File "D:\Program Files\Splunk\etc\apps\TA-workday\bin\ta_workday\splunktaucclib\rest_handler\handler.py", line 113, in wrapper\n for name, data, acl in meth(self, *args, **kwargs):\n File "D:\Program Files\Splunk\etc\apps\TA-workday\bin\ta_workday\splunktaucclib\rest_handler\handler.py", line 299, in _format_response\n masked = self.rest_credentials.decrypt_for_get(name, data)\n File "D:\Program Files\Splunk\etc\apps\TA-workday\bin\ta_workday\splunktaucclib\rest_handler\credentials.py", line 184, in decrypt_for_get\n clear_password = self._get(name)\n File "D:\Program Files\Splunk\etc\apps\TA-workday\bin\ta_workday\splunktaucclib\rest_handler\credentials.py", line 389, in _get\n string = mgr.get_password(user=context.username())\n File "D:\Program Files\Splunk\etc\apps\TA-workday\bin\ta_workday\solnlib\utils.py", line 154, in wrapper\n return func(*args, **kwargs)\n File "D:\Program Files\Splunk\etc\apps\TA-workday\bin\ta_workday\solnlib\credentials.py", line 118, in get_password\n all_passwords = self._get_all_passwords()\n File "D:\Program Files\Splunk\etc\apps\TA-workday\bin\ta_workday\solnlib\utils.py", line 154, in wrapper\n return func(*args, **kwargs)\n File "D:\Program Files\Splunk\etc\apps\TA-workday\bin\ta_workday\solnlib\credentials.py", line 272, in _get_all_passwords\n clear_password += field_clear[index]\nTypeError: cannot concatenate 'str' and 'NoneType' objects\n". See splunkd.log for more details.

11-07-2018 01:24:02.461 -0600 ERROR AdminManagerExternal - Unexpected error "" from python handler: "REST Error [500]: Internal Server Error -- Traceback (most recent call last):\n File "D:\Program Files\Splunk\etc\apps\TA-workday\bin\ta_workday\splunktaucclib\rest_handler\handler.py", line 113, in wrapper\n for name, data, acl in meth(self, *args, **kwargs):\n File "D:\Program Files\Splunk\etc\apps\TA-workday\bin\ta_workday\splunktaucclib\rest_handler\handler.py", line 299, in _format_response\n masked = self.rest_credentials.decrypt_for_get(name, data)\n File "D:\Program Files\Splunk\etc\apps\TA-workday\bin\ta_workday\splunktaucclib\rest_handler\credentials.py", line 184, in decrypt_for_get\n clear_password = self._get(name)\n File "D:\Program Files\Splunk\etc\apps\TA-workday\bin\ta_workday\splunktaucclib\rest_handler\credentials.py", line 389, in _get\n string = mgr.get_password(user=context.username())\n File "D:\Program Files\Splunk\etc\apps\TA-workday\bin\ta_workday\solnlib\utils.py", line 154, in wrapper\n return func(*args, **kwargs)\n File "D:\Program Files\Splunk\etc\apps\TA-workday\bin\ta_workday\solnlib\credentials.py", line 118, in get_password\n all_passwords = self._get_all_passwords()\n File "D:\Program Files\Splunk\etc\apps\TA-workday\bin\ta_workday\solnlib\utils.py", line 154, in wrapper\n return func(*args, **kwargs)\n File "D:\Program Files\Splunk\etc\apps\TA-workday\bin\ta_workday\solnlib\credentials.py", line 272, in _get_all_passwords\n clear_password += field_clear[index]\nTypeError: cannot concatenate 'str' and 'NoneType' objects\n". See splunkd.log for more details.

11-07-2018 01:24:02.651 -0600 ERROR ExecProcessor - message from "python "D:\Program Files\Splunk\etc\apps\TA-workday\bin\workday.py"" HTTPError: HTTP 500 Internal Server Error -- {"messages":[{"type":"ERROR","text":"Unexpected error \"\" from python handler: \"REST Error [500]: Internal Server Error -- Traceback (most recent call last):\n File \"D:\Program Files\Splunk\etc\apps\TA-workday\bin\ta_workday\splunktaucclib\rest_handler\handler.py\", line 113, in wrapper\n for name, data, acl in meth(self, *args, **kwargs):\n File \"D:\Program Files\Splunk\etc\apps\TA-workday\bin\ta_workday\splunktaucclib\rest_handler\handler.py\", line 299, in _format_response\n masked = self.rest_credentials.decrypt_for_get(name, data)\n File \"D:\Program Files\Splunk\etc\apps\TA-workday\bin\ta_workday\splunktaucclib\rest_handler\credentials.py\", line 184, in decrypt_for_get\n clear_password = self._get(name)\n File \"D:\Program Files\Splunk\etc\apps\TA-workday\bin\ta_workday\splunktaucclib\rest_handler\credentials.py\", line 389, in _get\n string = mgr.get_password(user=context.username())\n File \"D:\Program Files\Splunk\etc\apps\TA-workday\bin\ta_workday\solnlib\utils.py\", line 154, in wrapper\n return func(*args, **kwargs)\n File \"D:\Program Files\Splunk\etc\apps\TA-workday\bin\ta_workday\solnlib\credentials.py\", line 118, in get_password\n all_passwords = self._get_all_passwords()\n File \"D:\Program Files\Splunk\etc\apps\TA-workday\bin\ta_workday\solnlib\utils.py\", line 154, in wrapper\n return func(*args, **kwargs)\n File \"D:\Program Files\Splunk\etc\apps\TA-workday\bin\ta_workday\solnlib\credentials.py\", line 272, in _get_all_passwords\n clear_password += field_clear[index]\nTypeError: cannot concatenate 'str' and 'NoneType' objects\n\". See splunkd.log for more details."}]}

11-07-2018 01:24:03.173 -0600 ERROR ExecProcessor - message from "python "D:\Program Files\Splunk\etc\apps\TA-workday\bin\workday.py"" ERRORHTTP 500 Internal Server Error -- {"messages":[{"type":"ERROR","text":"Unexpected error \"\" from python handler: \"REST Error [500]: Internal Server Error -- Traceback (most recent call last):\n File \"D:\Program Files\Splunk\etc\apps\TA-workday\bin\ta_workday\splunktaucclib\rest_handler\handler.py\", line 113, in wrapper\n for name, data, acl in meth(self, *args, **kwargs):\n File \"D:\Program Files\Splunk\etc\apps\TA-workday\bin\ta_workday\splunktaucclib\rest_handler\handler.py\", line 299, in _format_response\n masked = self.rest_credentials.decrypt_for_get(name, data)\n File \"D:\Program Files\Splunk\etc\apps\TA-workday\bin\ta_workday\splunktaucclib\rest_handler\credentials.py\", line 184, in decrypt_for_get\n clear_password = self._get(name)\n File \"D:\Program Files\Splunk\etc\apps\TA-workday\bin\ta_workday\splunktaucclib\rest_handler\credentials.py\", line 389, in _get\n string = mgr.get_password(user=context.username())\n File \"D:\Program Files\Splunk\etc\apps\TA-workday\bin\ta_workday\solnlib\utils.py\", line 154, in wrapper\n return func(*args, **kwargs)\n File \"D:\Program Files\Splunk\etc\apps\TA-workday\bin\ta_workday\solnlib\credentials.py\", line 118, in get_password\n all_passwords = self._get_all_passwords()\n File \"D:\Program Files\Splunk\etc\apps\TA-workday\bin\ta_workday\solnlib\utils.py\", line 154, in wrapper\n return func(*args, **kwargs)\n File \"D:\Program Files\Splunk\etc\apps\TA-workday\bin\ta_workday\solnlib\credentials.py\", line 272, in _get_all_passwords\n clear_password += field_clear[index]\nTypeError: cannot concatenate 'str' and 'NoneType' objects\n\". See splunkd.log for more details."}]}

splunkaccess.log
127.0.0.1 - splunk-system-user [06/Nov/2018:15:48:45.336 -0600] "GET /servicesNS/nobody/TA-workday/properties/TA-workday HTTP/1.1" 404 143 - - - 1ms