Alerting

How to setup alert on multiple rows

cgiatras
Explorer

So I setup this search on an apache web log:

sourcetype="access_common" status=* | top status limit="1000"

Results are:

status count percent
200 250935 96.938500
302 3279 1.266708
404 1667 0.643977
500 1322 0.510701
401 819 0.316387
301 325 0.125550
400 261 0.100827
502 197 0.076103
206 43 0.016611
403 12 0.004636

Now I want to setup an alert that when the Percent Total for 2* and 3* percent falls below 95%. I will like this full report sent in the alert so not sure how to go about doing so. I know I can do :

sourcetype="access_common" status=* | top status limit="1000" | where status=2* OR status=3* but again I would like the above report in FULL when receiving the alert.

Thank you

Tags (2)
0 Karma
1 Solution

jonuwz
Influencer

You'll need to sum up the percentage of 'good' requests in a custom condition
i.e.

If your base search is

index=_internal source=*web_access.log | top status limit=1000 

Then you can do a custom condition of

appendpipe [ | stats sum(eval(if(match(status,"^[23]"),percent,0))) as percent | eval status="good"] | where status=="good" AND percent < 95

This adds another line to the results with a status of 'good' and a percent of the sum of all the 2* and 3* statuses, then if that percent is < 95, the alert triggers

Make sure to tick the box for 'Include results in email' and that the alert mode is 'once per search'

Update

If you find the goob / bad breakdown useful and want to include it in your report here's an alternative.

The main search becomes :

index=_internal source=*web_access.log 
| top status limit=1000     
| eval type=case(match(status,"^[23]"),"good",match(status,"^[45]"),"bad",1==1,"unknown")
| appendpipe [ stats sum(count) as count sum(percent) as percent by type 
               | rename type as status ] 
| fields - type

and the custom condition in the alert simply becomes :

where status=="good" AND percent < 95

View solution in original post

jonuwz
Influencer

You'll need to sum up the percentage of 'good' requests in a custom condition
i.e.

If your base search is

index=_internal source=*web_access.log | top status limit=1000 

Then you can do a custom condition of

appendpipe [ | stats sum(eval(if(match(status,"^[23]"),percent,0))) as percent | eval status="good"] | where status=="good" AND percent < 95

This adds another line to the results with a status of 'good' and a percent of the sum of all the 2* and 3* statuses, then if that percent is < 95, the alert triggers

Make sure to tick the box for 'Include results in email' and that the alert mode is 'once per search'

Update

If you find the goob / bad breakdown useful and want to include it in your report here's an alternative.

The main search becomes :

index=_internal source=*web_access.log 
| top status limit=1000     
| eval type=case(match(status,"^[23]"),"good",match(status,"^[45]"),"bad",1==1,"unknown")
| appendpipe [ stats sum(count) as count sum(percent) as percent by type 
               | rename type as status ] 
| fields - type

and the custom condition in the alert simply becomes :

where status=="good" AND percent < 95

cgiatras
Explorer

Thanks Jonuwz that looks great!

0 Karma

jonuwz
Influencer

updated with sample code for the good / bad stats in the report

0 Karma

cgiatras
Explorer

I came up with this:

sourcetype="access_common" status=* | top status limit="1000" | appendpipe [ | stats sum(eval(if(match(status,"^[23]"),percent,0))) as percent | eval status="good"] |appendpipe [ | stats sum(eval(if(match(status,"^[45]"),percent,0))) as percent | eval status="bad"]

0 Karma

cgiatras
Explorer

And what if I wanted to also get the total "good" and total "bad" in the report itself? So far its working great though it seems, just want to let it soak in for a few hours.

0 Karma

cgiatras
Explorer

Thanks Jonuwz . Will try this later today and let you know how it worked!

0 Karma
Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

Splunk is officially part of Cisco

Revolutionizing how our customers build resilience across their entire digital footprint.   Splunk ...

Splunk APM & RUM | Planned Maintenance March 26 - March 28, 2024

There will be planned maintenance for Splunk APM and RUM between March 26, 2024 and March 28, 2024 as ...