topic Re: Support Apache Tomcat Valves Extended Access Log in Getting Data In

Support Apache Tomcat Valves Extended Access Log

mldeschenes — Thu, 08 May 2014 19:25:52 GMT

I can't seem to get Splunk to auto/detect our current Apache Tomcat 6.x or 7.x logs.
Please help and appreciate the support, I have tried all I can so far. New to Splunk and not yet SME with this tool ... 🙂

Log source/format (Apache Tomcat 6.x – org.apache.catalina.valves.ExtendedAccessLogValve)

<Valve className="org.apache.catalina.valves.ExtendedAccessLogValve" directory="E:\folder-Logs" pattern="date time c-ip s-ip cs-method cs-uri-stem cs-uri-query sc-status bytes time-taken cs(User-Agent)    cs(Cookie) cs(Referer) cs(HOST)" prefix="${tomcat.instance.name}-" resolveHosts="false" suffix=".log"/>

Sample scrubbed http access log:

#Fields: date time c-ip s-ip cs-method cs-uri-stem cs-uri-query sc-status bytes time-taken cs(User-Agent)   cs(Cookie) cs(Referer) cs(HOST)
#Version: 2.0
#Software: Apache Tomcat/6.0.32
2014-05-06 04:04:09 7x.2xx.3x.5x 10.5x.7x.6x POST /folder/ajax/get.action - 200 79782 0.890 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_6_8) AppleWebKit/534.54.16 (KHTML, like Gecko) Version/5.1.4 Safari/534.54.16'    'JSESSIONID=BXA; CookiesEnabled=1; Sx7xFE=1xxxx.2xxxx.0000;' 'hxxps://client1.domain.com/folder/do.action?content=mypage=1' 'client1.skillport.com'

Re: Support Apache Tomcat Valves Extended Access Log

lguinn2 — Thu, 08 May 2014 23:25:04 GMT

I don't know what you mean by "autodetect", but this is the inputs.conf you probably need

[monitor://E:\folder-Logs]
sourcetype=access_combined_extended

For props.conf on the indexer, I would use

[access_combined_extended]
REPORT-ace=access_combined_base_fields
EXTRACT-aceExt1=\'(?<cs_User_Agent>.*?)\'.*?\'(?<cs_Cookie>.*?)\'.*?\'(?<cs_Referer>.*?)\'.*?\'(?<cs_Host>.*?)\'.
SHOULD_LINEMERGE = false
TIME_FORMAT=%Y-%m-%d %H:%M:%S
MAX_TIMESTAMP_LOOKAHEAD = 30

And for transforms.conf on the indexer

[access_combined_base_fields]
DELIMS = " "
FIELDS = date, time, c_ip, s_ip, cs_method, cs_uri_stem, cs_uri_query, sc_status, bytes, time_taken

Note: there shouldn't be any linebreak on the EXTRACT line above. Or the FIELDS line.

I just made up the sourcetype called access_combined_extended, because your data doesn't exactly match the common Apache formats I see. And I also set a few attributes in props.conf that you don't strictly need, but specifying them will help Splunk parse your data more efficiently.

Re: Support Apache Tomcat Valves Extended Access Log

mldeschenes — Fri, 09 May 2014 12:08:56 GMT

Appreciate the support, I am rather new to Splunk. Will give this a shot, is it possible to send me the files and I can simply copy/past? I'm assuming I simply need to modify existing files and add the info you provided?

Re: Support Apache Tomcat Valves Extended Access Log

mldeschenes — Fri, 09 May 2014 13:38:36 GMT

Sorry I can't seem to figure this out, please provide me exact files/path if all possible. I have fresh 6.1 install, don't care of any existing data as we are running poc/pilot.

Re: Support Apache Tomcat Valves Extended Access Log

lguinn2 — Fri, 09 May 2014 19:10:53 GMT

create each of the files named above in

$SPLUNK_HOME/etc/system/local

Probably only the inputs.conf file will already exist. But for any file that already exists, simply copy and paste the above at the end of the file.

After copying the files, then restart Splunk.

You should probably walk through the Splunk Tutorial at
http://docs.splunk.com/Documentation/Splunk/latest/SearchTutorial/WelcometotheSearchTutorial