Splunk Search

Why is scheduler failing after upgrading a search head pool?

ekost
Splunk Employee
Splunk Employee

Our search head pool nodes were recently upgraded from 6.6.1 to 7.3.0. After the upgrade, the scheduled searches have failed, breaking the reports and alerts. The log shows lots of empty warning messages:

12-24-2019 23:59:59.227 +0100 WARN SavedSplunker -

So DEBUG logging was enabled and we see messages like:

12-24-2019 23:59:59.226 +0100 DEBUG SavedSplunker - savedSearchUpdated field changed: savedsearch_id="admin;search;InternalSplunkdLogAlert", field_name="__ss_type", old_val="scheduled", new_val=""
12-24-2019 23:59:59.225 +0100 DEBUG SavedSplunker - AlertNotifier queued notifications=0, managedSearchCount=0, managedSchedulerSearchCount=0, managedSchedulerRTSearchCount=0
12-24-2019 23:59:59.224 +0100 DEBUG SavedSplunker - lock file already exists, search head skipping execution of: savedsearch_id="nobody;splunk_monitoring_console;DMC Asset - Build Standalone Asset Table", now=1562854500, 

The Search Head Pooling feature was deprecated for some time, but there was no mention of a bug or behavior change that I could find. What happened?

ekost
Splunk Employee
Splunk Employee

This is the known issue SPL-173038 that appears in Splunk 7.1.x, 7.2.x and 7.3.x effecting the Search Head Pooling (SHP) feature and the scheduler. No scheduled searches or alerts will run if SHP nodes are upgraded to 7.1.x and later. The only remediation is to switch to search head clustering. The search head pooling feature was removed in Splunk Enterprise 8.0.0.

Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...