I am having an issue getting our 'rest_ta' add on to run properly on a schedule. When I enter the data and 'save' it the call runs properly and data is added to the specified index. But, it does not run after that on the polling interval scheduled, whether this is a number of seconds or a cron scheduled. I also noticed when reloading that some of the values from the UI are not saved properly (for example, Polling Interval and Application Key). I don't think this issue is with the addon itself because if I use just the add on on a clean install everything works as expected. My guess is another add on is failing somewhere and blocking this from saving properly. When I click 'Save' the following shows up in the 'splunkd' log file. I don't see an error but was thinking if I could determine what order things are running in here I could look for the next app that should have run and try to remove/disable it to see if that fixes things.
07-17-2020 14:06:07.157 -0400 INFO ExecProcessor - New scheduled exec process: python /opt/splunk/etc/apps/TA-netapp_eseries/bin/rest.py
07-17-2020 14:06:07.157 -0400 INFO ExecProcessor - interval: run once
07-17-2020 14:06:07.157 -0400 INFO ExecProcessor - New scheduled exec process: python /opt/splunk/etc/apps/TA-netapp_eseries/bin/rest.py
07-17-2020 14:06:07.157 -0400 INFO ExecProcessor - interval: run once
07-17-2020 14:06:07.157 -0400 INFO ExecProcessor - New scheduled exec process: python /opt/splunk/etc/apps/TA-netapp_eseries/bin/rest.py
07-17-2020 14:06:07.157 -0400 INFO ExecProcessor - interval: run once
07-17-2020 14:06:07.157 -0400 INFO ExecProcessor - New scheduled exec process: python /opt/splunk/etc/apps/TA-netapp_eseries/bin/rest.py
07-17-2020 14:06:07.157 -0400 INFO ExecProcessor - interval: run once
07-17-2020 14:06:07.157 -0400 INFO ExecProcessor - New scheduled exec process: python /opt/splunk/etc/apps/TA-netapp_eseries/bin/rest.py
07-17-2020 14:06:07.157 -0400 INFO ExecProcessor - interval: run once
07-17-2020 14:06:07.157 -0400 INFO ExecProcessor - New scheduled exec process: python /opt/splunk/etc/apps/TA-netapp_eseries/bin/rest.py
07-17-2020 14:06:07.157 -0400 INFO ExecProcessor - interval: run once
07-17-2020 14:06:07.157 -0400 INFO ExecProcessor - New scheduled exec process: python /opt/splunk/etc/apps/TA-netapp_eseries/bin/rest.py
07-17-2020 14:06:07.157 -0400 INFO ExecProcessor - interval: run once
07-17-2020 14:06:07.157 -0400 INFO ExecProcessor - New scheduled exec process: python /opt/splunk/etc/apps/TA-netapp_eseries/bin/rest.py
07-17-2020 14:06:07.157 -0400 INFO ExecProcessor - interval: run once
07-17-2020 14:06:07.158 -0400 INFO ExecProcessor - New scheduled exec process: python /opt/splunk/etc/apps/TA-netapp_eseries/bin/rest.py
07-17-2020 14:06:07.158 -0400 INFO ExecProcessor - interval: run once
07-17-2020 14:06:07.158 -0400 INFO ExecProcessor - New scheduled exec process: python /opt/splunk/etc/apps/TA-netapp_eseries/bin/rest.py
07-17-2020 14:06:07.158 -0400 INFO ExecProcessor - interval: run once
07-17-2020 14:06:07.158 -0400 INFO ExecProcessor - New scheduled exec process: python /opt/splunk/etc/apps/TA-netapp_eseries/bin/rest.py
07-17-2020 14:06:07.158 -0400 INFO ExecProcessor - interval: run once
07-17-2020 14:06:07.158 -0400 INFO ExecProcessor - New scheduled exec process: python /opt/splunk/etc/apps/app-elections/bin/ftr_lookups.py
07-17-2020 14:06:07.158 -0400 INFO ExecProcessor - interval: run once
07-17-2020 14:06:07.158 -0400 INFO ExecProcessor - New scheduled exec process: /opt/splunk/bin/splunkd instrument-resource-usage
07-17-2020 14:06:07.158 -0400 INFO ExecProcessor - interval: 0 ms
07-17-2020 14:06:07.158 -0400 INFO ExecProcessor - New scheduled exec process: python /opt/splunk/etc/apps/netapp_app_eseries_perf/bin/eseries_folder_hierarchy_gen.py
07-17-2020 14:06:07.158 -0400 INFO ExecProcessor - interval: 60000 ms
07-17-2020 14:06:07.158 -0400 INFO ExecProcessor - New scheduled exec process: python /opt/splunk/etc/apps/search_activity/bin/CheckDataStats-events.py
07-17-2020 14:06:07.158 -0400 INFO ExecProcessor - interval: 600000 ms
07-17-2020 14:06:07.158 -0400 INFO ExecProcessor - New scheduled exec process: python /opt/splunk/etc/apps/search_activity/bin/CheckDataStats-search.py
07-17-2020 14:06:07.158 -0400 INFO ExecProcessor - interval: 600000 ms
07-17-2020 14:06:07.158 -0400 INFO ExecProcessor - interval="0 0 * * *" is a valid cron schedule
07-17-2020 14:06:07.158 -0400 INFO ExecProcessor - New scheduled exec process: python /opt/splunk/etc/apps/splunk_app_for_nix/bin/update_hosts.py
07-17-2020 14:06:07.158 -0400 INFO ExecProcessor - cron schedule: "0 0 * * *"
07-17-2020 14:06:07.158 -0400 INFO ExecProcessor - New scheduled exec process: python /opt/splunk/etc/apps/splunk_app_stream/bin/scripted_inputs/deploy_splunk_ta_stream.py
07-17-2020 14:06:07.158 -0400 INFO ExecProcessor - interval: run once
07-17-2020 14:06:07.158 -0400 INFO ExecProcessor - New scheduled exec process: python /opt/splunk/etc/apps/splunk_app_stream/bin/scripted_inputs/setup_independent_stream.py
07-17-2020 14:06:07.158 -0400 INFO ExecProcessor - interval: run once
07-17-2020 14:06:07.158 -0400 INFO ExecProcessor - interval="0 * * * *" is a valid cron schedule
07-17-2020 14:06:07.158 -0400 INFO ExecProcessor - New scheduled exec process: python /opt/splunk/etc/apps/splunk_instrumentation/bin/instrumentation.py
07-17-2020 14:06:07.158 -0400 INFO ExecProcessor - cron schedule: "0 * * * *"
07-17-2020 14:06:07.158 -0400 INFO ExecProcessor - New scheduled exec process: python /opt/splunk/etc/apps/splunk_instrumentation/bin/on_splunk_start.py
07-17-2020 14:06:07.158 -0400 INFO ExecProcessor - interval: run once
07-17-2020 14:06:07.158 -0400 INFO ExecProcessor - interval="0 0 * * *" is a valid cron schedule
07-17-2020 14:06:07.158 -0400 INFO ExecProcessor - New scheduled exec process: python /opt/splunk/etc/apps/splunk_instrumentation/bin/schedule_delete.py
07-17-2020 14:06:07.158 -0400 INFO ExecProcessor - cron schedule: "0 0 * * *"
07-17-2020 14:06:07.158 -0400 INFO ExecProcessor - New scheduled exec process: python /opt/splunk/etc/apps/splunk_monitoring_console/bin/dmc_config.py
07-17-2020 14:06:07.158 -0400 INFO ExecProcessor - interval: run once