Alerting

How to trigger alert based on result filed value count

Kwip
Contributor

I want to generate alert for the below query if it gives more than 10 results per minute for the continues 10 minutes. I can achieve it from the below query and setting the alert condition as search result equal to 10 and scheduling the alert to run on every 1 minute.

Query - eventtype=prod_servers sc_status!=200 | eval ComputerStatus=host."-"sc_status | bucket span=1m _time | stats count as TotalErrCount by _time ComputerStatus | where TotalErrCount >10 | stats value(ComputerStatus ) as ErrorCode count by _time

But the problem is the eval field ComputerStatus comprises of many values (I.e, server1-404, server2-404, server1-500, server3-403.,,,) so in 10 minutes span I am getting result like as below

02:27 10:01-server2-404
02:27 10:02-server2-404
02:27 10:03-server1-500
02:27 10:04-server1-500
02:27 10:05-server1-500
02:27 10:06-server1-500
02:27 10:07-server3-403
02:27 10:08-server3-403
02:27 10:09-server3-403
02:27 10:10-server2-404

My alert conditions are satisfied for the above results and alert getting triggered. but I want the alert to be generated if any one of the filed value count is greater than 10 per 1 minute for the continuous 10 minutes.
Example
02:27 10:01-server2-404 OR server1-500
02:27 10:02-server2-404 OR server1-500
02:27 10:03-server2-404 OR server1-500
02:27 10:04-server2-404 OR server1-500
02:27 10:05-server2-404 OR server1-500
02:27 10:06-server2-404 OR server1-500
02:27 10:07-server2-404 OR server1-500
02:27 10:08-server2-404 OR server1-500
02:27 10:09-server2-404 OR server1-500
02:27 10:10-server2-404 OR server1-500

Tags (1)
0 Karma
1 Solution

pradeepkumarg
Influencer

I replied on your other thread. Posting same here

eventtype=prd_servers sc_status!=300 sc_status!=200 sc_status!=0 | eval computerstatus=host:"-":sc_status | bin span=1m _time  | stats count by _time, computerstatus | search count>10 | eventstats count as occurences by computerstatus | search occurences > 10

View solution in original post

0 Karma

pradeepkumarg
Influencer

I replied on your other thread. Posting same here

eventtype=prd_servers sc_status!=300 sc_status!=200 sc_status!=0 | eval computerstatus=host:"-":sc_status | bin span=1m _time  | stats count by _time, computerstatus | search count>10 | eventstats count as occurences by computerstatus | search occurences > 10
0 Karma

Kwip
Contributor

Thank you!

0 Karma
Get Updates on the Splunk Community!

Welcome to the Splunk Community!

(view in My Videos) We're so glad you're here! The Splunk Community is place to connect, learn, give back, and ...

Tech Talk | Elevating Digital Service Excellence: The Synergy of Splunk RUM & APM

Elevating Digital Service Excellence: The Synergy of Real User Monitoring and Application Performance ...

Adoption of RUM and APM at Splunk

    Unleash the power of Splunk Observability   Watch Now In this can't miss Tech Talk! The Splunk Growth ...