https://community.splunk.com/t5/Deployment-Architecture/Scheduled-Search-not-triggered-with-message-Defer-search-quot/m-p/397390#M14393 <P>Issue turned out to be - Search Head bind to two cluster Master , one was upgraded to 7.1 and other was not. Once the CM that was not upgraded was removed- all searches worked fine. </P> <P>@server.conf <BR /> [clustering]<BR /> master_uri = CM1:8089,CM2:8089<BR /> mode = searchhead</P> Mon, 14 May 2018 00:33:08 GMT rbal_splunk 2018-05-14T00:33:08Z https://community.splunk.com/t5/Deployment-Architecture/Scheduled-Search-not-triggered-with-message-Defer-search-quot/m-p/397389#M14392 <P>We have upgraded Cluster Master and Cluster Peer and Search head to splunk version 7.1 . The search head shows error like below</P> <PRE><CODE>Indexer Clustering: Search summary - AD logins by username, workstation created by jon-scheidell on the search app was deferred to run after the searchable rolling restart or upgrade is completed. There are currently 7825 deferred searches in total. 5/9/2018, 4:01:00 PM Indexer Clustering: Search DMC Asset - Build Standalone Asset Table created by nobody on the splunk_monitoring_console app was skipped during the searchable rolling restart or upgrade. 5/9/2018, 4:01:00 PM </CODE></PRE> <P>Other Observation : <BR /> i)none of the Scheduled searches are running.<BR /> ii) The scheduler.log on the Search head shows messages like below </P> <PRE><CODE>INFO SavedSplunker - Skip search "DMC Asset - Build Standalone Asset Table" during searchable rolling process with nextRunTime=1525884660 INFO SavedSplunker - Defer search "summary - AD logins by username, workstation" during searchable rolling process INFO SavedSplunker - Defer search "Summary - firewall drops by product" during searchable rolling process INFO SavedSplunker - Defer search "Summary - WAF blocks" during searchable rolling process INFO SavedSplunker - Defer search "Summary - Maximo ODTicketDump Deduped" during searchable rolling process INFO SavedSplunker - Skip search "Optiv Populate Lookups - bambenekIPs - dest_ip" during searchable rolling process with nextRunTime=1525884720 </CODE></PRE> <P>…</P> <P>III) The search Head’s splunkd.log file spammed with these message</P> <PRE><CODE>CMMessages - Master Splunk version is not supported with this version of cluster master. CMMessages - Searchable rolling restart is not supported with this version of cluster master. For more information, see the documentation on searchable rolling restart. </CODE></PRE> <P>iv)The Cluster Master’s splunkd.log file has errors like</P> <PRE><CODE>WARN CMMessages - Searchable rolling restart is not supported with this version of cluster master. For more information, see the documentation on searchable rolling restart. WARN CMMessages - Master Splunk version is not supported with this version of cluster master. INFO ClusterMasterControlHandler - Request rejected. Searchable rolling restart cannot be guaranteed if search factor=1. </CODE></PRE> Mon, 14 May 2018 00:28:27 GMT https://community.splunk.com/t5/Deployment-Architecture/Scheduled-Search-not-triggered-with-message-Defer-search-quot/m-p/397389#M14392 rbal_splunk 2018-05-14T00:28:27Z https://community.splunk.com/t5/Deployment-Architecture/Scheduled-Search-not-triggered-with-message-Defer-search-quot/m-p/397390#M14393 <P>Issue turned out to be - Search Head bind to two cluster Master , one was upgraded to 7.1 and other was not. Once the CM that was not upgraded was removed- all searches worked fine. </P> <P>@server.conf <BR /> [clustering]<BR /> master_uri = CM1:8089,CM2:8089<BR /> mode = searchhead</P> Mon, 14 May 2018 00:33:08 GMT https://community.splunk.com/t5/Deployment-Architecture/Scheduled-Search-not-triggered-with-message-Defer-search-quot/m-p/397390#M14393 rbal_splunk 2018-05-14T00:33:08Z