Alerting

How to search Apache access logs and alert if status code 500 appears for more than 10 minutes?

krishnacasso
Path Finder

I am new to Splunk. We will be using it to monitor our Apache logs. I need to configure an alert for the Apache access log. If the 500 status in the access log appears more than 10 minutes I need get a alert saying the Apache services were down.

xx.xx.xx.xx - - [2/Feb/2016:12:44:05 -0600] 7765 "GET /forms/login.fcc HTTP/1.1" 200 3094 "-" "-"
xx.xx.xx.xx - - [2/Feb/2016:12:44:06 -0600] 8704 "GET /forms/login.fcc HTTP/1.1" 200 3094 "-" "-"
xx.xx.xx.xx - - [2/Feb/2016:12:44:06 -0600] 12730 "GET /forms/login.fcc HTTP/1.1" 200 3094 "-" "-"
xx.xx.xx.xx - - [2/Feb/2016:12:44:07 -0600] 10192 "GET /forms/login.fcc HTTP/1.1" 200 3094 "-" "-"
xx.xx.xx.xx - - [2/Feb/2016:12:44:07 -0600] 8749 "GET /forms/login.fcc HTTP/1.1" 500 3094 "-" "-"
xx.xx.xx.xx - - [2/Feb/2016:12:44:07 -0600] 8222 "GET /forms/login.fcc HTTP/1.1" 500 3094 "-" "-"
xx.xxx.xx.xx - - [2/Feb/2016:12:44:08 -0600] 9021 "GET /forms/login.fcc HTTP/1.1" 500 3094 "-" "-"
xx.xxx.xx.xx - - [2/Feb/2016:12:44:08 -0600] 9306 "GET /forms/login.fcc HTTP/1.1" 500 3094 "-" "-"
xx.xxx.xx.xx - - [2/Feb/2016:12:44:08 -0600] 8598 "GET /forms/login.fcc HTTP/1.1" 500 3094 "-" "-"
xx.xxx.xx.xx - - [2/Feb/2016:12:44:09 -0600] 8482 "GET /forms/login.fcc HTTP/1.1" 200 3094 "-" "-"
xx.xxx.xx.xx - - [2/Feb/2016:12:44:09 -0600] 8647 "GET /forms/login.fcc HTTP/1.1" 200 3094 "-" "-"
xx.xxx.xx.xx - - [2/Feb/2016:12:44:09 -0600] 8750 "GET /forms/login.fcc HTTP/1.1" 200 3094 "-" "-"

I need to get an alert if 500 appears for more than 10 minutes.
Can you help with writing a search for this?

Thanks.

0 Karma

MuS
SplunkTrust
SplunkTrust

Hi krishnacasso,

Welcome to Splunk 🙂

if you already have a field called status which represents the status code, you can run a search like this:

your base search to get the apache logs earliest=-11min@min | bin _time span=1min | stats count(eval(status=="500")) by time | where count>=10

This will count status="500" per minute and if the count is bigger or equal 10 show the result. You can safe this as an alert http://docs.splunk.com/Documentation/Splunk/6.3.3/Alert/Aboutalerts

If the status field is not yet available run the above search with some regex to get the field :

your base search to get the apache logs earliest=-11min@min | rex ""\s(?<status>\d+)\s\d" | bin _time span=1min | stats count(eval(status=="500")) by time | where count>=10

Hope this helps to get you started ...

cheers, MuS

PS: This is un-tested, because I don't have a Splunk handy right now 😉

Got questions? Get answers!

Join the Splunk Community Slack to learn, troubleshoot, and make connections with fellow Splunk practitioners in real time!

Meet up IRL or virtually!

Join Splunk User Groups to connect and learn in-person by region or remotely by topic or industry.

Get Updates on the Splunk Community!

Index This | What travels the world but is also stuck in place?

April 2026 Edition  Hayyy Splunk Education Enthusiasts and the Eternally Curious!   We’re back with this ...

Discover New Use Cases: Unlock Greater Value from Your Existing Splunk Data

Realizing the full potential of your Splunk investment requires more than just understanding current usage; it ...

Continue Your Journey: Join Session 2 of the Data Management and Federation Bootcamp ...

As data volumes continue to grow and environments become more distributed, managing and optimizing data ...