Alerting

How to generate alerts based on the result count per minute over a specific time of span?

Kwip
Contributor

Below is my search.

eventtype=prd_servers sc_status!=300 sc_status!=200 sc_status!=0 | eval computerstatus=host:"-":sc_status | stats count by computerstatus | search count>10

I want to trigger alert for the above search for the below condition:

Condition: Results should be above 10 per minute for continuous 5 minutes.

Tags (2)
0 Karma
1 Solution

pradeepkumarg
Influencer
eventtype=prd_servers sc_status!=300 sc_status!=200 sc_status!=0 | eval computerstatus=host:"-":sc_status | bin span=1m _time  | stats count by _time | search count>10
earliest = -6m@m
latest=-1m@m

Alert trigger
condition - if number of events
is equal to = 5

You can schedule this to run every 3 mins

View solution in original post

pradeepkumarg
Influencer
eventtype=prd_servers sc_status!=300 sc_status!=200 sc_status!=0 | eval computerstatus=host:"-":sc_status | bin span=1m _time  | stats count by _time | search count>10
earliest = -6m@m
latest=-1m@m

Alert trigger
condition - if number of events
is equal to = 5

You can schedule this to run every 3 mins

pradeepkumarg
Influencer

Updated the search with correct field in stats count

0 Karma

Kwip
Contributor

I want to generate alert for the below query if it gives more than 10 results per minute for the continues 10 minutes. I can achieve it from the below query and setting the alert condition as search result equal to 10 and scheduling the alert to run on every 1 minute.

Query - eventtype=prod_servers sc_status!=200 | eval ComputerStatus=host."-"sc_status | bucket span=1m _time | stats count as TotalErrCount by _time ComputerStatus | where TotalErrCount >10 | stats value(ComputerStatus ) as ErrorCode count by _time

But the problem is the eval field ComputerStatus comprises of many values (I.e, server1-404, server2-404, server1-500, server3-403.,,,) so in 10 minutes span I am getting result like as below

02:27 10:01-server2-404
02:27 10:02-server2-404
02:27 10:03-server1-500
02:27 10:04-server1-500
02:27 10:05-server1-500
02:27 10:06-server1-500
02:27 10:07-server3-403
02:27 10:08-server3-403
02:27 10:09-server3-403
02:27 10:10-server2-404

My alert conditions are satisfied for the above results and alert getting triggered. but I want the alert to be generated if any one of the filed value count is greater than 10 per 1 minute for the continuous 10 minutes.
Example
02:27 10:01-server2-404 OR server1-500
02:27 10:02-server2-404 OR server1-500
02:27 10:03-server2-404 OR server1-500
02:27 10:04-server2-404 OR server1-500
02:27 10:05-server2-404 OR server1-500
02:27 10:06-server2-404 OR server1-500
02:27 10:07-server2-404 OR server1-500
02:27 10:08-server2-404 OR server1-500
02:27 10:09-server2-404 OR server1-500
02:27 10:10-server2-404 OR server1-500

0 Karma

pradeepkumarg
Influencer

Something like below shoud work for you

eventtype=prd_servers sc_status!=300 sc_status!=200 sc_status!=0 | eval computerstatus=host:"-":sc_status | bin span=1m _time  | stats count by _time, computerstatus | search count>10 | eventstats count as occurences by computerstatus | search occurences > 10
0 Karma

Kwip
Contributor

Thank you! It works.

0 Karma
Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

Splunk is officially part of Cisco

Revolutionizing how our customers build resilience across their entire digital footprint.   Splunk ...

Splunk APM & RUM | Planned Maintenance March 26 - March 28, 2024

There will be planned maintenance for Splunk APM and RUM between March 26, 2024 and March 28, 2024 as ...