How to create summary report out of Apache logs?

[himanshuarora7]

Trying to put all the required info from Apache logs in one spreadsheet.

I'm trying to pull out uri_path, min/max and avg response time duration, total hits, hit counts on all the status codes and referer. Trying to sum up the count as per status code (say - total hit counts on 404 and 200) based on uri_path.

Please help me with search command for this?

[kristian_kolb]

You can't really do all those things in one search, but to give you a hint (change field names as needed;

sourcetype=access_combined | stats c min(response_time) max(response_time) avg(response_time) by uri_path 


sourcetype=access_combined | chart c over status by uri_path

You should probably read up on the docs pages for the stats and chart commands, as well as the User manual;

http://docs.splunk.com/Documentation/Splunk/5.0.2/Search/Whatsinthismanual
http://docs.splunk.com/Documentation/Splunk/5.0.2/SearchReference/Chart
http://docs.splunk.com/Documentation/Splunk/5.0.2/SearchReference/Stats

Hope this helps,

Kristian

[himanshuarora7]

Yeah, i see "%T" for apache LogFormat seems missing, i'll have that corrected.

thx

[kristian_kolb]

Neither could I. If you want to calculate durations or anything that depends on that field (time_taken, response_time, exec_time or whatever you choose to call it), you'll have to configure the logging.

[himanshuarora7]

Ideally yes, the response time should be at the end of every log event but i couldn't see it.

Here is the sample log event from the webserver.

68.150.108.25 - - [26/Mar/2013:15:00:16 -0500] "GET /application/com/comps/desk/common/desktop_common_header_a/libs/img/bg.jpg?01AD=3_x8iDneE-BJxZRd2cZpzJobRrUMaIArqjOJZLidrsjiyN7vGZBEeVw&01RI=C1E12A8647E8301&01NA= HTTP/1.1" 200 10069 "http://testurl.com/software/desk.html" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_3) AppleWebKit/536.28.10 (KHTML, like Gecko) Version/6.0.3 Safari/536.28.10"

[kristian_kolb]

You should post a few sample log events. Probably the response time is near the end of each line, if it's logged at all.

You should ask your web admin regarding what is being logged. s/he should be able to tell you which field is the response time. Or to enable logging of response time it not yet configured that way.

/k

[himanshuarora7]

I have extracted the required fields already but couldn't find the field for response time duration, may be it's there but i'm not sure of the exact name. Can you please let me know the exact name of the field i should search for?

[yannK]

make sure that you extract those fields, or simply use the correct sourcetype (access_combined) that comes with the automatic fields extractions.

[himanshuarora7]

Alright, i'll look into the doc pages for this, can you help me find the fields for duration or max,min response_time, because when i put them in search it shows the blank fields.

I couldn't find the fields for min,max duration or the response time. Do i have to enable them somewhere? Can you please help me with that?