Alerting

How do I setup an alert that triggers and take action on second search?

antmob
New Member

I have an alert that triggers when the search returns 0 events for the last couple of hours and sends a slack message. It runs every 5 minutes on cron and looks a few hour back. However, for some reason that I don't know, the alert false triggers some times when it should not and when I manually do the search for which it triggered, I see a bunch of events during that time span. This happens once a month or so.

So if anyone know a solution for this or why this happens that would be great. If not, I'm thinking of changing the alert so that it only triggers if the result is 0 for 2 searches in a row (5 minutes in between) to avoid the false triggers. Is it possible to do this and how?

Labels (3)
0 Karma

Nisha18789
Builder

Hi @antmob 

For investigating the issue, you might want to start with checking the actual search window of the false alert instance using query like :

index=_internal sourcetype=scheduler status=success savedsearch_name=<name of your alert>

And, for 2nd question on 2 consecutive - 0 results, you can join the results of below query with your alert logic or you can use summary index to store the results and join with summary indexed data.

index=_internal sourcetype=scheduler status=success savedsearch_name=<name of your alert> result_count=0

Hope this helps!

 

 

 

 

0 Karma
Register for .conf21 Now! Go Vegas or Go Virtual!

How will you .conf21? You decide! Go in-person in Las Vegas, 10/18-10/21, or go online with .conf21 Virtual, 10/19-10/20.