Alerting

How to search Apache access logs and alert if status code 500 appears for more than 10 minutes?

krishnacasso
Path Finder

I am new to Splunk. We will be using it to monitor our Apache logs. I need to configure an alert for the Apache access log. If the 500 status in the access log appears more than 10 minutes I need get a alert saying the Apache services were down.

xx.xx.xx.xx - - [2/Feb/2016:12:44:05 -0600] 7765 "GET /forms/login.fcc HTTP/1.1" 200 3094 "-" "-"
xx.xx.xx.xx - - [2/Feb/2016:12:44:06 -0600] 8704 "GET /forms/login.fcc HTTP/1.1" 200 3094 "-" "-"
xx.xx.xx.xx - - [2/Feb/2016:12:44:06 -0600] 12730 "GET /forms/login.fcc HTTP/1.1" 200 3094 "-" "-"
xx.xx.xx.xx - - [2/Feb/2016:12:44:07 -0600] 10192 "GET /forms/login.fcc HTTP/1.1" 200 3094 "-" "-"
xx.xx.xx.xx - - [2/Feb/2016:12:44:07 -0600] 8749 "GET /forms/login.fcc HTTP/1.1" 500 3094 "-" "-"
xx.xx.xx.xx - - [2/Feb/2016:12:44:07 -0600] 8222 "GET /forms/login.fcc HTTP/1.1" 500 3094 "-" "-"
xx.xxx.xx.xx - - [2/Feb/2016:12:44:08 -0600] 9021 "GET /forms/login.fcc HTTP/1.1" 500 3094 "-" "-"
xx.xxx.xx.xx - - [2/Feb/2016:12:44:08 -0600] 9306 "GET /forms/login.fcc HTTP/1.1" 500 3094 "-" "-"
xx.xxx.xx.xx - - [2/Feb/2016:12:44:08 -0600] 8598 "GET /forms/login.fcc HTTP/1.1" 500 3094 "-" "-"
xx.xxx.xx.xx - - [2/Feb/2016:12:44:09 -0600] 8482 "GET /forms/login.fcc HTTP/1.1" 200 3094 "-" "-"
xx.xxx.xx.xx - - [2/Feb/2016:12:44:09 -0600] 8647 "GET /forms/login.fcc HTTP/1.1" 200 3094 "-" "-"
xx.xxx.xx.xx - - [2/Feb/2016:12:44:09 -0600] 8750 "GET /forms/login.fcc HTTP/1.1" 200 3094 "-" "-"

I need to get an alert if 500 appears for more than 10 minutes.
Can you help with writing a search for this?

Thanks.

0 Karma

MuS
SplunkTrust
SplunkTrust

Hi krishnacasso,

Welcome to Splunk 🙂

if you already have a field called status which represents the status code, you can run a search like this:

your base search to get the apache logs earliest=-11min@min | bin _time span=1min | stats count(eval(status=="500")) by time | where count>=10

This will count status="500" per minute and if the count is bigger or equal 10 show the result. You can safe this as an alert http://docs.splunk.com/Documentation/Splunk/6.3.3/Alert/Aboutalerts

If the status field is not yet available run the above search with some regex to get the field :

your base search to get the apache logs earliest=-11min@min | rex ""\s(?<status>\d+)\s\d" | bin _time span=1min | stats count(eval(status=="500")) by time | where count>=10

Hope this helps to get you started ...

cheers, MuS

PS: This is un-tested, because I don't have a Splunk handy right now 😉

Get Updates on the Splunk Community!

Routing logs with Splunk OTel Collector for Kubernetes

The Splunk Distribution of the OpenTelemetry (OTel) Collector is a product that provides a way to ingest ...

Welcome to the Splunk Community!

(view in My Videos) We're so glad you're here! The Splunk Community is place to connect, learn, give back, and ...

Tech Talk | Elevating Digital Service Excellence: The Synergy of Splunk RUM & APM

Elevating Digital Service Excellence: The Synergy of Real User Monitoring and Application Performance ...