How do I setup an alert that triggers and take action on second search?

New Member

I have an alert that triggers when the search returns 0 events for the last couple of hours and sends a slack message. It runs every 5 minutes on cron and looks a few hour back. However, for some reason that I don't know, the alert false triggers some times when it should not and when I manually do the search for which it triggered, I see a bunch of events during that time span. This happens once a month or so.

So if anyone know a solution for this or why this happens that would be great. If not, I'm thinking of changing the alert so that it only triggers if the result is 0 for 2 searches in a row (5 minutes in between) to avoid the false triggers. Is it possible to do this and how?

Labels (3)
0 Karma


Hi @antmob 

For investigating the issue, you might want to start with checking the actual search window of the false alert instance using query like :

index=_internal sourcetype=scheduler status=success savedsearch_name=<name of your alert>

And, for 2nd question on 2 consecutive - 0 results, you can join the results of below query with your alert logic or you can use summary index to store the results and join with summary indexed data.

index=_internal sourcetype=scheduler status=success savedsearch_name=<name of your alert> result_count=0

Hope this helps!





0 Karma
Get Updates on the Splunk Community!

BSides Splunk 2022 - The Call for Papers is now Open!

TLDR; Main Site: CFP Site: CFP Opens: December 15th, ...

Sending Metrics to Splunk Enterprise With the OpenTelemetry Collector

This blog post is part of an ongoing series on OpenTelemetry. The OpenTelemetry project is the second largest ...

What's New in Splunk Cloud Platform 9.0.2208?!

Howdy!  We are happy to share the newest updates in Splunk Cloud Platform 9.0.2208! Analysts can benefit ...