topic Re: How can we edit the regular expressions to extract fields from our Apache Access logs? in All Apps and Add-ons

How can we edit the regular expressions to extract fields from our Apache Access logs?

romangarth — Thu, 08 Dec 2016 18:26:03 GMT

We just installed Splunk this week (YAY!) and are trying to get our Apache logs digesting to start building dashboards. Our web store uses a modified Apache Access log format that looks like this in our www.conf:

%h %l %u %t %V \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\" %U %q

I know that none of the extractions provided by access_combined or apache:access (Apache Addon) seem to work because they're looking for the default format, but I'm a little confused on how to help them recognize our format. I see the extractions listed, but they don't look like regular regexes that I've seen. How would I go about changing them to match our formatting?

Appreciate any help!

Re: How can we edit the regular expressions to extract fields from our Apache Access logs?

AnthonyTibaldi — Mon, 12 Dec 2016 19:33:06 GMT

What is the sourcetype for your logs? Are any of the fields being extracted when you search on the sourcetype?

Re: How can we edit the regular expressions to extract fields from our Apache Access logs?

romangarth — Mon, 12 Dec 2016 21:00:46 GMT

The sourcetype I've tried apache:access (From the Apache Addon) and access_combined (from the default install), and neither extract data or fields properly.

Re: How can we edit the regular expressions to extract fields from our Apache Access logs?

koshyk — Mon, 12 Dec 2016 21:02:51 GMT

I can see the default extraction in the TA is:

EXTRACT-apache_access= ^(?<src>[^ ]+)\s+(?<site>.+)\s+(?<ident>\S+)\s+(?<user>\S+)\s+(?<dest_port>\d+)\s+\[(?<timestamp>\d+/\w+/\d+:\d+:\d+:\d+\s+[-+]\d+)[^"\n]*"(?<request>[^"]+)[^ \n]*\s+\"(?<uri_query>[^ ]*)\"\s+(?<status>\d+)(?:[^ \n]* ){2}\"(?<http_referer>[^"]+)\"\s+\"(?<http_user_agent>[^"]+)\"\s+(?<request_bytes>\d+)\s+(?<response_bytes>\d+)\s+(?<response_time_microseconds>\d+)

This matches with your pattern to a certain degree, but NOT all.
(eg src matches to %h ; Unless you put the actual log, we can't be sure which all matches)
So you might need to amend this "EXTRACT-apache_access" in your "local" directory of the app (or your own app) with the www.conf