Alerting

create alert when http error code from a given IP reaches a threshold

cscchen
New Member

Hi,

I'd like to create alert for the following scenario:

in http access log, we have different return codes 2xx,3xx,4xx,5xx, etc. I'd like to get an alert when within a 10 minutes windows, the returned code 4xx for an source IP is higher than 10%. The alert will include all the source IPs that meet the percentage requirement as well as the percentage of 4xx. The alert would show sth. like the following:

source ip percentage of 4xx
a.b.c.d 20%
x.x.x.x 13%

One difficulty here is that new IPs keep showing up. Would really appreciate if anyone can help.

Thanks.

Tags (2)
0 Karma

chris
Motivator

Hi,

you could use a search similar to this (the status field contains the http status code an host would contain the ip for your example):

sourcetype="access_combined" | eval status_category=substr(status,0,1)+"xx" | stats count as total count(eval(status_category="4xx")) as bad_status by host | eval Percentage=100*round(bad_status/total,2) | where Percentage>10

Schedule the search to run in realtime over a window of 10 minutes and create one alert per result.

If you need help creating the alert let me know. If you need help with the search -> post some example events and let me know if the source ip & status are extracted as fields (if you do not know what a field is yet I'll explain).

0 Karma
Got questions? Get answers!

Join the Splunk Community Slack to learn, troubleshoot, and make connections with fellow Splunk practitioners in real time!

Meet up IRL or virtually!

Join Splunk User Groups to connect and learn in-person by region or remotely by topic or industry.

Get Updates on the Splunk Community!

Mile High Learning with Splunk University, Denver, Colorado

If Denver is known for its mile-high elevation, Splunk University is about to raise the bar on technical ...

IT Service Intelligence 5.0 Series: Your Guide to the June Launch

We are excited to announce the June release of Splunk IT Service Intelligence (ITSI) 5.0. This update ...

Agent Mode Engaged! Enchaining Agentic Operations with Splunk AI Assistant 2.0

    Are you ready to transform how your team handles complex data requests? We invite you to our upcoming ...