Why is scheduler failing after upgrading a search head pool?

[ekost]

Our search head pool nodes were recently upgraded from 6.6.1 to 7.3.0. After the upgrade, the scheduled searches have failed, breaking the reports and alerts. The log shows lots of empty warning messages:

12-24-2019 23:59:59.227 +0100 WARN SavedSplunker -

So DEBUG logging was enabled and we see messages like:

12-24-2019 23:59:59.226 +0100 DEBUG SavedSplunker - savedSearchUpdated field changed: savedsearch_id="admin;search;InternalSplunkdLogAlert", field_name="__ss_type", old_val="scheduled", new_val=""
12-24-2019 23:59:59.225 +0100 DEBUG SavedSplunker - AlertNotifier queued notifications=0, managedSearchCount=0, managedSchedulerSearchCount=0, managedSchedulerRTSearchCount=0
12-24-2019 23:59:59.224 +0100 DEBUG SavedSplunker - lock file already exists, search head skipping execution of: savedsearch_id="nobody;splunk_monitoring_console;DMC Asset - Build Standalone Asset Table", now=1562854500, 

The Search Head Pooling feature was deprecated for some time, but there was no mention of a bug or behavior change that I could find. What happened?

[ekost]

This is the known issue SPL-173038 that appears in Splunk 7.1.x, 7.2.x and 7.3.x effecting the Search Head Pooling (SHP) feature and the scheduler. No scheduled searches or alerts will run if SHP nodes are upgraded to 7.1.x and later. The only remediation is to switch to search head clustering. The search head pooling feature was removed in Splunk Enterprise 8.0.0.