Our search head pool nodes were recently upgraded from 6.6.1 to 7.3.0. After the upgrade, the scheduled searches have failed, breaking the reports and alerts. The log shows lots of empty warning messages:
This is the known issue SPL-173038 that appears in Splunk 7.1.x, 7.2.x and 7.3.x effecting the Search Head Pooling (SHP) feature and the scheduler. No scheduled searches or alerts will run if SHP nodes are upgraded to 7.1.x and later. The only remediation is to switch to search head clustering. The search head pooling feature was removed in Splunk Enterprise 8.0.0.