As I said above we are seeing this issue with Real-Time and Scheduled, I have the issue happening right now witha a Real-Time search in my test environment. I am firing test events into an existing index, I can see them arriving in the index, if I run the search in the alert as a manual search it shows the event but the the action isn't triggering and when I run index="_internal" sourcetype!=splunkd_remote_searches savedsearch savedsearch_name=NOC_Alcatel result_count=1 I see hits from my first few test alerts but nothing from after it stopped responding. Yes we are running a 3 node cluster. I'm not seeing any issues in DMC the skip ratio over the last 4 hours is 0% I am seeing these every minute 08-26-2021 10:59:01.230 +0100 INFO SHCMaster - delegate search job requested for savedsearch_name="NOC_Alcatel" host = pr-l-sphead-03v.dcnlab.sset.local source = /opt/splunk/var/log/splunk/splunkd.log sourcetype = splunkd 08-26-2021 10:59:01.230 +0100 INFO SHCMaster - realtime search savedsearch_name=NOC_Alcatel selector=nobody;noc;NOC_Alcatel sid=rt_scheduler__admin__noc__RMD5aff146651f76f3cb_at_1629386520_7_5E69A627-7FCE-4CAE-A9C4-E72E65CBF04A is either already running or being dispatched. Ignoring request. host = pr-l-sphead-03v.dcnlab.sset.local source = /opt/splunk/var/log/splunk/splunkd.log sourcetype = splunkd but I am also seeing these going back forever so I dont think they are anything to do with this issue
... View more