Splunk Search

How can I create an alert that will detect a behavior change in count of events that persist for a certain amount of time?

jedatt01
Builder

I'm trying to create an alert that will trigger when the count of events is changed drastically from one time bucket to another and that change persists for a certain amount of time....say 30 minutes. I have been able to detect a change using streamstats, but I'm getting stuck on how to detect when the change persists for a certain amount of time.

The following command works to detect the change...

streamstats window=2 global=f current=t first(count) As p_count

But how do I expand that to detect a persistent change over more than two time buckets?

example:

time        event         count
9:00 AM MY_EVENT 0
9:05 AM MY_EVENT 75
9:10 AM MY_EVENT 90
9:15 AM MY_EVENT 80
9:20 AM MY_EVENT 85
9:25 AM MY_EVENT 75
9:30 AM MY_EVENT 95 <------------ I want alert to trigger here
9:35 AM MY_EVENT 74
9:40 AM MY_EVENT 0 

But I don't want to trigger an event if I see the following..

time        event         count
9:00 AM MY_EVENT 0
9:05 AM MY_EVENT 75
9:10 AM MY_EVENT 80
9:15 AM MY_EVENT 0
9:20 AM MY_EVENT 0
9:25 AM MY_EVENT 0
9:30 AM MY_EVENT 0 
9:35 AM MY_EVENT 0
9:40 AM MY_EVENT 0
0 Karma

sundareshr
Legend

Would something like this work?

... | table time event count | timechart span=30min count sum(eval(if(count>0, 1. 0))) as persist by event | where count=persist
0 Karma

jedatt01
Builder

I get where you are coming from with this, but my example table was probably too simplistic. There is a possibility that the normal value of MY_EVENT is not 0. I need a way to detect a consecutive series of abnormal values for count.

0 Karma
Get Updates on the Splunk Community!

Splunk Decoded: Service Maps vs Service Analyzer Tree View vs Flow Maps

It’s Monday morning, and your phone is buzzing with alert escalations – your customer-facing portal is running ...

What’s New in Splunk Observability – September 2025

What's NewWe are excited to announce the latest enhancements to Splunk Observability, designed to help ITOps ...

Fun with Regular Expression - multiples of nine

Fun with Regular Expression - multiples of nineThis challenge was first posted on Slack #regex channel ...