Tracing Response time of apache using splunk

[vpraveenindia]

I am tracking the response time(in seconds) of the pages served by apache using "%T" and i would like to track all the request which are taking more than "10" seconds to serve to the clients. Please let me know if there is a way to achieve.

[gcoles · Accepted solution]

In your case the time (in seconds) field comes three positions after the request details. Since the request is always enclosed in quotes in apache logs, I like to use them as an anchor point for the regex, rather than the start of the log message (it allows for a simpler regex). Also, you have three numeric fields following a double-quote and preceding another double-quote, so I would write a regex like this:

\" \d+ \d+ (?P< response_time >\d+(?:\.\d+)?)

Note that this forum won't let me use the correct syntax above, so if you use this regex with either rex or the field extraction UI, remove the spaces on either side of response_time. I am not sure if the %T field may include decimal points or not, so I made the regex support them optionally.

You can use the regex with either rex (inline pipe to rex), or you can save this field extraction for your apache sourcetype (Manager > Fields > Field Extractions).

Here is what a search might look like with Field Extraction configured:

index=xxx sourcetype=access_combined response_time>10

And with inline rex:

index=xxx sourcetype=access_combined | rex "\" \d+ \d+ (?P< response_time >\d+(?:\.\d+)?)" | where response_time>10

I should note that you also have the %D parameter in your apache configuration, which is the response time in microseconds. I would personally use this field as it has better resolution and its at the end of the log, which makes it easier to anchor within the string:

(?P< response_time >\d+)$

Then just adjust your where/search syntax for the precision, response_time>10000000.

As I mentioned before, make sure to remove the spaces around 'response_time' above, when you use it in any of the splunk UI's.

[vpraveenindia]

You are simply awesome!

[vpraveenindia]

Apologies for the delay.. below is the snippet from the apache log and the Log format

10.0.0.2 - - [03/Jan/2013:13:50:31 +0000] "GET /admin HTTP/1.1" 200 293 0 "-" "http://test.test.com" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.1; Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; .NET4.0C; .NET4.0E; InfoPath.2)" 1338

"%h %l %u %t \"%r\" %>s %b %T \"%{LoggedIn}e\" \"%{Referer}i\" \"%{User-Agent}i\" %D"

[gcoles]

Do you already have a Field Extraction set up for the response_time field? If you do, you can simply use greater and less than signs in the search:

index=weblogs response_time>10

[gcoles]

No problem, can you paste in a sample log record or your apache LogFormat directive?Apache log files typically don't include field=value pairs (just values), so regular expressions must be position-sensitive (eg populate this field with the fifth element in the message).

[vpraveenindia]

Thanks for your reply, unfortunately i didn't have that setup already, would you be able to help me with that please? may be a small hint will do.