Alerting

Alert when count is less than a certain threshold

cpraznowski_spl
Splunk Employee
Splunk Employee

Hi...here is my search:

sourcetype="isc:dhcp"   earliest=-10m@s latest=now |    stats count  as dhcp_count by _time | where dhcp_count<5000

I'll usually get returned stats:
4800
10,000
11,000

I have this running on 5 minute cron schedule...don't see results. Perhaps I shall try real-time every 5 minutes...any thoughts appreciated

0 Karma
1 Solution

somesoni2
Revered Legend

Try this

alert search

sourcetype="isc:dhcp"   earliest=-11m@m latest=-1m@m|    stats count  as dhcp_count by _time | where dhcp_count<5000

Alert schedule - Cron

1-59/5 * * * *

Alert condition

When number of events > 0

View solution in original post

0 Karma

somesoni2
Revered Legend

Try this

alert search

sourcetype="isc:dhcp"   earliest=-11m@m latest=-1m@m|    stats count  as dhcp_count by _time | where dhcp_count<5000

Alert schedule - Cron

1-59/5 * * * *

Alert condition

When number of events > 0
0 Karma

cpraznowski_spl
Splunk Employee
Splunk Employee

woohoo, thanks a ton

0 Karma

woodcock
Esteemed Legend

You have not specified the most important part: what is your timepicker? Typically, this range is similar to the period so maybe you are running with Last 5 minutes which maybe is not a long enough timespan to ever rack up 5000 hits. When you say you "usually get ...", what was your timepicker value?

0 Karma

woodcock
Esteemed Legend

Surely you do not have 11K events per second, right? Your BY _time clause seems silly to me; probably you should be doing this instead:

 sourcetype="isc:dhcp" earliest=-10m@s latest=now | stats count AS dhcp_count | where dhcp_count<5000

And I would put the | where ... part inside your alert threshold settings.

0 Karma

cpraznowski_spl
Splunk Employee
Splunk Employee

Not quite....11K events per 5 minute period

..And I would put the | where ... part inside your alert threshold settings.... that's what I think I need to try..thanks

0 Karma

cpraznowski_spl
Splunk Employee
Splunk Employee

thanks ! 🙂

0 Karma

cpraznowski_spl
Splunk Employee
Splunk Employee

Last ten minutes: earliest=-10m@s latest=now

0 Karma

jkat54
SplunkTrust
SplunkTrust

Why do you have the "by clause" in your stats command, or a stats command at all? Shouldnt it just be:

 sourcetype="isc:dhcp" earliest=-10m@s latest=now | where count<5000

See if that helps.

0 Karma

cpraznowski_spl
Splunk Employee
Splunk Employee

by clause because the dchp server sends logs every 5 minutes
12:10 4800
12:15 10000
12:20 12000

The span is 10 minutes in period, so I want to alert when any of the _time dimensions < 5000

0 Karma

somesoni2
Revered Legend

Don't do real-time scheduled alert as it never ends. When you say your don't see result mean that alert is not triggered?

0 Karma

cpraznowski_spl
Splunk Employee
Splunk Employee

hi...yes, thanks for responding, I get the results above in my stats table, but when I access, Activity, Triggered alerts (search all)...I do not see results. In the above example, I was expecting to see the alert triggered b/c the count was 4,800...ie <5000

0 Karma

somesoni2
Revered Legend

The result that you gave, is it from the search which includes the where clause? Can you check in the scheduler logs to see what is the result_count returned by the search?

index=_internal sourcetype=scheduler savedsearch_name="YourAlertSearchName"  
0 Karma

cpraznowski_spl
Splunk Employee
Splunk Employee

cool, will take a look

0 Karma
Get Updates on the Splunk Community!

How to Monitor Google Kubernetes Engine (GKE)

We’ve looked at how to integrate Kubernetes environments with Splunk Observability Cloud, but what about ...

Index This | How can you make 45 using only 4?

October 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with this ...

Splunk Education Goes to Washington | Splunk GovSummit 2024

If you’re in the Washington, D.C. area, this is your opportunity to take your career and Splunk skills to the ...