Alerting

How to generate alerts based on the result count per minute over a specific time of span?

Kwip
Contributor

Below is my search.

eventtype=prd_servers sc_status!=300 sc_status!=200 sc_status!=0 | eval computerstatus=host:"-":sc_status | stats count by computerstatus | search count>10

I want to trigger alert for the above search for the below condition:

Condition: Results should be above 10 per minute for continuous 5 minutes.

Tags (2)
0 Karma
1 Solution

pradeepkumarg
Influencer
eventtype=prd_servers sc_status!=300 sc_status!=200 sc_status!=0 | eval computerstatus=host:"-":sc_status | bin span=1m _time  | stats count by _time | search count>10
earliest = -6m@m
latest=-1m@m

Alert trigger
condition - if number of events
is equal to = 5

You can schedule this to run every 3 mins

View solution in original post

pradeepkumarg
Influencer
eventtype=prd_servers sc_status!=300 sc_status!=200 sc_status!=0 | eval computerstatus=host:"-":sc_status | bin span=1m _time  | stats count by _time | search count>10
earliest = -6m@m
latest=-1m@m

Alert trigger
condition - if number of events
is equal to = 5

You can schedule this to run every 3 mins

pradeepkumarg
Influencer

Updated the search with correct field in stats count

0 Karma

Kwip
Contributor

I want to generate alert for the below query if it gives more than 10 results per minute for the continues 10 minutes. I can achieve it from the below query and setting the alert condition as search result equal to 10 and scheduling the alert to run on every 1 minute.

Query - eventtype=prod_servers sc_status!=200 | eval ComputerStatus=host."-"sc_status | bucket span=1m _time | stats count as TotalErrCount by _time ComputerStatus | where TotalErrCount >10 | stats value(ComputerStatus ) as ErrorCode count by _time

But the problem is the eval field ComputerStatus comprises of many values (I.e, server1-404, server2-404, server1-500, server3-403.,,,) so in 10 minutes span I am getting result like as below

02:27 10:01-server2-404
02:27 10:02-server2-404
02:27 10:03-server1-500
02:27 10:04-server1-500
02:27 10:05-server1-500
02:27 10:06-server1-500
02:27 10:07-server3-403
02:27 10:08-server3-403
02:27 10:09-server3-403
02:27 10:10-server2-404

My alert conditions are satisfied for the above results and alert getting triggered. but I want the alert to be generated if any one of the filed value count is greater than 10 per 1 minute for the continuous 10 minutes.
Example
02:27 10:01-server2-404 OR server1-500
02:27 10:02-server2-404 OR server1-500
02:27 10:03-server2-404 OR server1-500
02:27 10:04-server2-404 OR server1-500
02:27 10:05-server2-404 OR server1-500
02:27 10:06-server2-404 OR server1-500
02:27 10:07-server2-404 OR server1-500
02:27 10:08-server2-404 OR server1-500
02:27 10:09-server2-404 OR server1-500
02:27 10:10-server2-404 OR server1-500

0 Karma

pradeepkumarg
Influencer

Something like below shoud work for you

eventtype=prd_servers sc_status!=300 sc_status!=200 sc_status!=0 | eval computerstatus=host:"-":sc_status | bin span=1m _time  | stats count by _time, computerstatus | search count>10 | eventstats count as occurences by computerstatus | search occurences > 10
0 Karma

Kwip
Contributor

Thank you! It works.

0 Karma
Get Updates on the Splunk Community!

Infographic provides the TL;DR for the 2024 Splunk Career Impact Report

We’ve been buzzing with excitement about the recent validation of Splunk Education! The 2024 Splunk Career ...

Enterprise Security Content Update (ESCU) | New Releases

In December, the Splunk Threat Research Team had 1 release of new security content via the Enterprise Security ...

Why am I not seeing the finding in Splunk Enterprise Security Analyst Queue?

(This is the first of a series of 2 blogs). Splunk Enterprise Security is a fantastic tool that offers robust ...