Deployment Architecture
Highlighted

Scheduled Search not triggered with message Defer search "summary - AD logins by username, workstation" during searchable rolling

Splunk Employee
Splunk Employee

We have upgraded Cluster Master and Cluster Peer and Search head to splunk version 7.1 . The search head shows error like below

Indexer Clustering: Search summary - AD logins by username, workstation created by jon-scheidell on the search app was deferred to run after the searchable rolling restart or upgrade is completed. There are currently 7825 deferred searches in total.
5/9/2018, 4:01:00 PM
Indexer Clustering: Search DMC Asset - Build Standalone Asset Table created by nobody on the splunk_monitoring_console app was skipped during the searchable rolling restart or upgrade.
5/9/2018, 4:01:00 PM

Other Observation :
i)none of the Scheduled searches are running.
ii) The scheduler.log on the Search head shows messages like below

INFO SavedSplunker - Skip search "DMC Asset - Build Standalone Asset Table" during searchable rolling process with nextRunTime=1525884660 
 INFO SavedSplunker - Defer search "summary - AD logins by username, workstation" during searchable rolling process 
 INFO SavedSplunker - Defer search "Summary - firewall drops by product" during searchable rolling process 
 INFO SavedSplunker - Defer search "Summary - WAF blocks" during searchable rolling process 
 INFO SavedSplunker - Defer search "Summary - Maximo ODTicketDump Deduped" during searchable rolling process 
 INFO SavedSplunker - Skip search "Optiv Populate Lookups - bambenekIPs - dest_ip" during searchable rolling process with nextRunTime=1525884720 

III) The search Head’s splunkd.log file spammed with these message

CMMessages - Master Splunk version is not supported with this version of cluster master.
CMMessages - Searchable rolling restart is not supported with this version of cluster master. For more information, see the documentation on searchable rolling restart.

iv)The Cluster Master’s splunkd.log file has errors like

WARN CMMessages - Searchable rolling restart is not supported with this version of cluster master. For more information, see the documentation on searchable rolling restart.
WARN CMMessages - Master Splunk version is not supported with this version of cluster master.
INFO ClusterMasterControlHandler - Request rejected. Searchable rolling restart cannot be guaranteed if search factor=1.
0 Karma
Highlighted

Re: Scheduled Search not triggered with message Defer search "summary - AD logins by username, workstation" during searchable rolling

Splunk Employee
Splunk Employee

Issue turned out to be - Search Head bind to two cluster Master , one was upgraded to 7.1 and other was not. Once the CM that was not upgraded was removed- all searches worked fine.

@server.conf
[clustering]
master_uri = CM1:8089,CM2:8089
mode = searchhead