REST API Modular Input issues after upgrading

[williamchenyp]

This is affecting one of our HF that we use to do ingest external data via scripts, vendor provided apps and REST API polls.   For the REST API part we use the REST API Modular Input app (https://splunkbase.splunk.com/app/1546/).  The REST inputs works without any issues when we were at Splunk Enterprise 7.1.3.

After upgrade SE to 8.1.1 and the rest_ta app to 2.0.1 last weekend, none of the scheduled REST inputs worked.   Problem is, this only happens on this server.   The REST inputs still work on a separate, dev server that was also upgraded to SE 8.1.1 and rest_ta 2.0.1. 

I see the following set of error events in splunkd.log but they only show up when I make a change to any of the REST inputs, like changing the cron schedule to force it to run at the next minute.

 Exception in thread Thread-1:
 Traceback (most recent call last):
   File "/opt/splunk/lib/python3.7/threading.py", line 926, in _bootstrap_inner
     self.run()
   File "/opt/splunk/lib/python3.7/threading.py", line 870, in run
     self._target(*self._args, **self._kwargs)
   File "/opt/splunk/etc/apps/rest_ta/bin/rest.py", line 447, in do_run
     endpoint_list[i] = endpoint.replace(replace_key,c['clear_password'])
   File "/opt/splunk/lib/python3.7/site-packages/splunk/entity.py", line 574, in __getitem__
     return self.properties[key]
 KeyError: 'clear_password'

 I do not see any errors at the times when the cron schedules's supposed to execute the API calls.   So it feels like the rest_ta app itself just quit working.  Honestly, I'm a bit lost trying to interpret the errors.  Anyone have seen something similar, or have any tips on how to resolve this?

I tried removing the app completely, restart splunkd then reinstall and reconfigure rest_ta 2.0.1 from scratch.  Still none of the scheduled jobs run.  The same errors still only show up after I modified one of the REST inputs.  

Here's one of the several REST inputs configured.   They're all identical in that I'm only using the bundled "JSONArrayHandler" response_handler to process the returning JSON data from Infoblox.  It's not customized in any way.

 [rest://InfoBlox_Networks]
 activation_key = --snip--
 auth_password = {encrypted:splunk_svc_user}
 auth_type = basic
 auth_user = splunk_svc_user
 delimiter = :
 endpoint = https://a.b.c.d/wapi/v2.6.1/network?_max_results=15000
 host = a.b.c.d
 http_method = GET
 index = infoblox
 index_error_response_codes = 1
 log_level = INFO
 polling_interval = 3 * * * *
 request_timeout = 60
 response_handler = JSONArrayHandler
 response_type = json
 sequential_mode = 0
 sourcetype = infoblox:api:network
 streaming_request = 0