Alerting

How to alert when statusCode=403 is over X percent of total traffic, and greater than a count of 100?

Filmhooligan
New Member

I have an alert that checks for a percentage of requests that are 403'ing:

index=myIndex "POST /url1" OR "POST /url2 "
| stats count by statusCode
| eventstats sum(count) as percent 
| eval percent=round(count*100/percent,2) 
| fields percent,statusCode
| search (statusCode="403")
| search percent > 2

I'm hoping to add a condition for when traffic is slow and percentages might be skewed. How can I add a condition for the alert to fire only if the percent is > 2% of all traffic AND the amount of 403's is greater than 100?

Thanks for your help!

0 Karma
1 Solution

gokadroid
Motivator

How about changing these lines of your query | search (statusCode="403")| search percent > 2 and making them as follows to see if it works for your case:

...| search (statusCode="403" AND count>=100 AND percent > 2)

View solution in original post

0 Karma

vinothkumark
Path Finder

Hi, can you help on the query if multiple condition needs to be met in the same query? 
Example: status code is 500 and greater than 10% alert should be triggered. also, if status code is 403 and greater than 20% alert should be triggered.

0 Karma

gokadroid
Motivator

How about changing these lines of your query | search (statusCode="403")| search percent > 2 and making them as follows to see if it works for your case:

...| search (statusCode="403" AND count>=100 AND percent > 2)
0 Karma

Filmhooligan
New Member

Thanks for your help! Unfortunately that did not work. I tuned down my numbers to verify that the search was doing what I wanted, down to count>=1 AND percent>.5 I got "No results found" when running that, but when running my search with just the percent changed to | search percent > 2 I got a result of 0.55%

0 Karma

gokadroid
Motivator

Did you add count in this line of SPL | fields percent,statusCode

Unless you include field count in it to make it | fields count, percent,statusCode you will not be able to search on it in
| search (statusCode="403" AND count>=100 AND percent > 2)

0 Karma

Filmhooligan
New Member

Oh yeah that makes sense, this works just how I needed it now, thank you!

0 Karma
Get Updates on the Splunk Community!

Upcoming Webinar: Unmasking Insider Threats with Slunk Enterprise Security’s UEBA

Join us on Wed, Dec 10. at 10AM PST / 1PM EST for a live webinar and demo with Splunk experts! Discover how ...

.conf25 technical session recap of Observability for Gen AI: Monitoring LLM ...

If you’re unfamiliar, .conf is Splunk’s premier event where the Splunk community, customers, partners, and ...

A Season of Skills: New Splunk Courses to Light Up Your Learning Journey

There’s something special about this time of year—maybe it’s the glow of the holidays, maybe it’s the ...