Alerting

How do I setup an alert that triggers and take action on second search?

antmob
New Member

I have an alert that triggers when the search returns 0 events for the last couple of hours and sends a slack message. It runs every 5 minutes on cron and looks a few hour back. However, for some reason that I don't know, the alert false triggers some times when it should not and when I manually do the search for which it triggered, I see a bunch of events during that time span. This happens once a month or so.

So if anyone know a solution for this or why this happens that would be great. If not, I'm thinking of changing the alert so that it only triggers if the result is 0 for 2 searches in a row (5 minutes in between) to avoid the false triggers. Is it possible to do this and how?

Labels (3)
0 Karma

Nisha18789
Builder

Hi @antmob 

For investigating the issue, you might want to start with checking the actual search window of the false alert instance using query like :

index=_internal sourcetype=scheduler status=success savedsearch_name=<name of your alert>

And, for 2nd question on 2 consecutive - 0 results, you can join the results of below query with your alert logic or you can use summary index to store the results and join with summary indexed data.

index=_internal sourcetype=scheduler status=success savedsearch_name=<name of your alert> result_count=0

Hope this helps!

 

 

 

 

0 Karma
Get Updates on the Splunk Community!

This Week's Community Digest - Splunk Community Happenings [9.26.22]

Get the latest news and updates from the Splunk Community here! Upcoming User Group Events! &#x1f44f; Check ...

BSides Splunk 2022 - The Call for Papers is now Open!

TLDR; Main Site: https://bsidessplunk.com CFP Site: https://bsidessplunk.com/cfp CFP Opens: December 15th, ...

Sending Metrics to Splunk Enterprise With the OpenTelemetry Collector

This blog post is part of an ongoing series on OpenTelemetry. The OpenTelemetry project is the second largest ...