Alerting

How to alert when statusCode=403 is over X percent of total traffic, and greater than a count of 100?

Filmhooligan
New Member

I have an alert that checks for a percentage of requests that are 403'ing:

index=myIndex "POST /url1" OR "POST /url2 "
| stats count by statusCode
| eventstats sum(count) as percent 
| eval percent=round(count*100/percent,2) 
| fields percent,statusCode
| search (statusCode="403")
| search percent > 2

I'm hoping to add a condition for when traffic is slow and percentages might be skewed. How can I add a condition for the alert to fire only if the percent is > 2% of all traffic AND the amount of 403's is greater than 100?

Thanks for your help!

0 Karma
1 Solution

gokadroid
Motivator

How about changing these lines of your query | search (statusCode="403")| search percent > 2 and making them as follows to see if it works for your case:

...| search (statusCode="403" AND count>=100 AND percent > 2)

View solution in original post

0 Karma

gokadroid
Motivator

How about changing these lines of your query | search (statusCode="403")| search percent > 2 and making them as follows to see if it works for your case:

...| search (statusCode="403" AND count>=100 AND percent > 2)
0 Karma

Filmhooligan
New Member

Thanks for your help! Unfortunately that did not work. I tuned down my numbers to verify that the search was doing what I wanted, down to count>=1 AND percent>.5 I got "No results found" when running that, but when running my search with just the percent changed to | search percent > 2 I got a result of 0.55%

0 Karma

gokadroid
Motivator

Did you add count in this line of SPL | fields percent,statusCode

Unless you include field count in it to make it | fields count, percent,statusCode you will not be able to search on it in
| search (statusCode="403" AND count>=100 AND percent > 2)

0 Karma

Filmhooligan
New Member

Oh yeah that makes sense, this works just how I needed it now, thank you!

0 Karma
Get Updates on the Splunk Community!

Routing Data to Different Splunk Indexes in the OpenTelemetry Collector

This blog post is part of an ongoing series on OpenTelemetry. The OpenTelemetry project is the second largest ...

Getting Started with AIOps: Event Correlation Basics and Alert Storm Detection in ...

Getting Started with AIOps:Event Correlation Basics and Alert Storm Detection in Splunk IT Service ...

Register to Attend BSides SPL 2022 - It's all Happening October 18!

Join like-minded individuals for technical sessions on everything Splunk!  This is a community-led and run ...