Splunk Search

Why is scheduler failing after upgrading a search head pool?

ekost
Splunk Employee
Splunk Employee

Our search head pool nodes were recently upgraded from 6.6.1 to 7.3.0. After the upgrade, the scheduled searches have failed, breaking the reports and alerts. The log shows lots of empty warning messages:

12-24-2019 23:59:59.227 +0100 WARN SavedSplunker -

So DEBUG logging was enabled and we see messages like:

12-24-2019 23:59:59.226 +0100 DEBUG SavedSplunker - savedSearchUpdated field changed: savedsearch_id="admin;search;InternalSplunkdLogAlert", field_name="__ss_type", old_val="scheduled", new_val=""
12-24-2019 23:59:59.225 +0100 DEBUG SavedSplunker - AlertNotifier queued notifications=0, managedSearchCount=0, managedSchedulerSearchCount=0, managedSchedulerRTSearchCount=0
12-24-2019 23:59:59.224 +0100 DEBUG SavedSplunker - lock file already exists, search head skipping execution of: savedsearch_id="nobody;splunk_monitoring_console;DMC Asset - Build Standalone Asset Table", now=1562854500, 

The Search Head Pooling feature was deprecated for some time, but there was no mention of a bug or behavior change that I could find. What happened?

ekost
Splunk Employee
Splunk Employee

This is the known issue SPL-173038 that appears in Splunk 7.1.x, 7.2.x and 7.3.x effecting the Search Head Pooling (SHP) feature and the scheduler. No scheduled searches or alerts will run if SHP nodes are upgraded to 7.1.x and later. The only remediation is to switch to search head clustering. The search head pooling feature was removed in Splunk Enterprise 8.0.0.

Get Updates on the Splunk Community!

Splunk Education - Fast Start Program!

Welcome to Splunk Education! Splunk training programs are designed to enable you to get started quickly and ...

Five Subtly Different Ways of Adding Manual Instrumentation in Java

You can find the code of this example on GitHub here. Please feel free to star the repository to keep in ...

New Splunk APM Enhancements Help Troubleshoot Your MySQL and NoSQL Databases Faster

Splunk Observability has two new enhancements to make it quicker and easier to troubleshoot slow or frequently ...