topic Re: Alert randomly stops working in Alerting

Alert randomly stops working

mscomms — Tue, 24 Aug 2021 10:34:01 GMT

Hi All,

I am seeing a strange issue where occaisionally one of my alerts stop working ( not always the same one ). When this issue is happening I can see the searches running but there are no triggers happening for the alert even when manually running the search finds the events.

I have tweaked the searches to make sure I am not falling foul of the _indextime vs _time issue caused by alerts arriving outside the search window.

It appears that the search just stops triggering and it starts again when I Disable/Enable the search.

Anyone else seeing this or have any ideas?

Re: Alert randomly stops working

burwell — Tue, 24 Aug 2021 22:32:06 GMT

Hi .. under alert type do you have Scheduled or Real-time?

Also, not sure if you have a search head cluster and a monitoring console/DMC. That might show you skipped searches etc.

Finally there are records for the saved search in the audit log. This might help you see if the the scheduler believes that the jobs were completed.

Re: Alert randomly stops working

mscomms — Wed, 25 Aug 2021 07:56:33 GMT

Wer are seeing the same issue with both but are in the process of migrating from Real-Time to scheduled as the scheduled gives faster results

Re: Alert randomly stops working

mscomms — Thu, 26 Aug 2021 13:07:02 GMT

As I said above we are seeing this issue with Real-Time and Scheduled, I have the issue happening right now witha a Real-Time search in my test environment.

I am firing test events into an existing index, I can see them arriving in the index, if I run the search in the alert as a manual search it shows the event but the the action isn't triggering and when I run

index="_internal" sourcetype!=splunkd_remote_searches savedsearch savedsearch_name=NOC_Alcatel result_count=1

I see hits from my first few test alerts but nothing from after it stopped responding.

Yes we are running a 3 node cluster.

I'm not seeing any issues in DMC the skip ratio over the last 4 hours is 0%

I am seeing these every minute

08-26-2021 10:59:01.230 +0100 INFO SHCMaster - delegate search job requested for savedsearch_name="NOC_Alcatel"

host = pr-l-sphead-03v.dcnlab.sset.local
source = /opt/splunk/var/log/splunk/splunkd.log
sourcetype = splunkd

08-26-2021 10:59:01.230 +0100 INFO SHCMaster - realtime search savedsearch_name=NOC_Alcatel selector=nobody;noc;NOC_Alcatel sid=rt_scheduler__admin__noc__RMD5aff146651f76f3cb_at_1629386520_7_5E69A627-7FCE-4CAE-A9C4-E72E65CBF04A is either already running or being dispatched. Ignoring request.

host = pr-l-sphead-03v.dcnlab.sset.local
source = /opt/splunk/var/log/splunk/splunkd.log
sourcetype = splunkd

but I am also seeing these going back forever so I dont think they are anything to do with this issue

Re: Alert randomly stops working

mscomms — Fri, 27 Aug 2021 09:07:22 GMT

I see nothing relating to this search in the audit.log