Alerting

Alert on |search field=0 returns too many rows?

bmgilmore
Path Finder

I've got a scheduled search that calculates the variability of a numeric field over time that should always be moving. If I pipe a "search Variability=0" at the end, and run in search view, the search runs, shows a lot of rows at first as it calculates back through time, and then shows the correct number of rows.

Oddly, when I set this search to alarm, it sends many alarms and the CSV search results attached show many (almost all) of the available rows, each with a Variability of 0. Returning to the triggered job returns 0 (or whatever 1 or 2) rows as I would expect. Especially strange is that each time the alert triggers, it is a new number of false positive rows.

It's almost as if the search is not waiting to complete before triggering the alert? Any ideas here? Thanks in advance!

Tags (1)
0 Karma

jonuwz
Influencer

You probably need a custom condition in your alert

i.e.

main search

... | stats count(eval(Variability==0)) as not_variable

then a custom condition where

search not_variable > 0
0 Karma

MHibbin
Influencer

attached? ... you will need to paste it in the question as code (using the button "101010", or by starting each line of the code with 4 spaces, or by inclosing in backticks).

0 Karma
Get Updates on the Splunk Community!

Updated Team Landing Page in Splunk Observability

We’re making some changes to the team landing page in Splunk Observability, based on your feedback. The ...

New! Splunk Observability Search Enhancements for Splunk APM Services/Traces and ...

Regardless of where you are in Splunk Observability, you can search for relevant APM targets including service ...

Webinar Recap | Revolutionizing IT Operations: The Transformative Power of AI and ML ...

The Transformative Power of AI and ML in Enhancing Observability   In the realm of IT operations, the ...