Alerting

create alert when http error code from a given IP reaches a threshold

cscchen
New Member

Hi,

I'd like to create alert for the following scenario:

in http access log, we have different return codes 2xx,3xx,4xx,5xx, etc. I'd like to get an alert when within a 10 minutes windows, the returned code 4xx for an source IP is higher than 10%. The alert will include all the source IPs that meet the percentage requirement as well as the percentage of 4xx. The alert would show sth. like the following:

source ip percentage of 4xx
a.b.c.d 20%
x.x.x.x 13%

One difficulty here is that new IPs keep showing up. Would really appreciate if anyone can help.

Thanks.

Tags (2)
0 Karma

chris
Motivator

Hi,

you could use a search similar to this (the status field contains the http status code an host would contain the ip for your example):

sourcetype="access_combined" | eval status_category=substr(status,0,1)+"xx" | stats count as total count(eval(status_category="4xx")) as bad_status by host | eval Percentage=100*round(bad_status/total,2) | where Percentage>10

Schedule the search to run in realtime over a window of 10 minutes and create one alert per result.

If you need help creating the alert let me know. If you need help with the search -> post some example events and let me know if the source ip & status are extracted as fields (if you do not know what a field is yet I'll explain).

0 Karma
Got questions? Get answers!

Join the Splunk Community Slack to learn, troubleshoot, and make connections with fellow Splunk practitioners in real time!

Meet up IRL or virtually!

Join Splunk User Groups to connect and learn in-person by region or remotely by topic or industry.

Get Updates on the Splunk Community!

Monitoring AI Agents with Splunk Observability Cloud

Let’s say I’m running a travel planning AI app in production. A user asks for three concise hotel options in ...

[Puzzles] Solve, Learn, Repeat: Tiling

This puzzle (first published here) is based on finding groups of tessellated tiles (inspired by floor tiles I ...

SOK it to Me: Top 3 Benefits of Using Splunk Operator on Kubernetes that’ll Make ...

    Thursday, July 9, 2026  |  11:00AM–12:00PM PDT Duration: 1 hour (includes Q&A) Managing can feel like a ...