Getting Data In

How to handle a logfile with two timestamps

Champion

Hi,

I have a catalina.out logfile that starts with one timestamp (YYYY-MM-DD HH:MM:SS), and then, after the initialization is complete, starts using a different one (MM.DD.YYYY HH:MM:SS). The customer needs all events, so I can't filter out the startup. How would this get handled?

2014-11-02 04:15:52 [cf6fb877] info    [native] dynaTrace Bootstrap Agent 5.5.0.5226
2014-11-02 04:15:52 [cf6fb877] info    [native] Using /fidcn/install/dynatrace-5.5.0 as Agent storage
2014-11-02 04:15:52 [cf6fb877] info    [native] Log file at /fidcn/install/dynatrace-5.5.0/log/dt_CTG_FidelityCentral_prod_bootstra
p_4145.log
2014-11-02 04:15:52 [cf6fb877] info    [native] Trying to connect to dtcoll-ctgprod-mmk.fmr.com:9998 for up to 60 seconds
2014-11-02 04:15:52 [cf6fb877] info    [native] Connected successfully
...
03.11.2014 06:22:45 ERROR [org.apache.jetspeed.container.JetspeedContainerServlet.doGet():350] JetspeedContainerServlet: Error rendering portlet "fcapps::FCStats": javax.portlet.PortletException: Request processing failed
javax.portlet.PortletException: Request processing failed
        at org.springframework.web.portlet.FrameworkPortlet.processRequest(FrameworkPortlet.java:544)
        at org.springframework.web.portlet.FrameworkPortlet.doDispatch(FrameworkPortlet.java:470)
        at javax.portlet.GenericPortlet.render(GenericPortlet.java:253)
        at org.apache.jetspeed.portlet.PortletObjectProxy.proxyRender(PortletObjectProxy.java:280)
        at org.apache.jetspeed.portlet.PortletObjectProxy.invoke(PortletObjectProxy.java:172)
        at $Proxy179.render(Unknown Source)
        at org.apache.jetspeed.factory.JetspeedPortletInstance.render(JetspeedPortletInstance.java:117)
        at org.apache.jetspeed.container.services.JetspeedFilterChain.doFilter(JetspeedFilterChain.java:142)
        at org.apache.jetspeed.container.services.JetspeedFilterChain.processFilter(JetspeedFilterChain.java:92)
        at org.apache.jetspeed.container.services.JetspeedFilterManager.processFilter(JetspeedFilterManager.java:112)
        at org.apache.jetspeed.container.JetspeedContainerServlet.doGet(JetspeedContainerServlet.java:290)
Tags (2)
0 Karma
1 Solution

SplunkTrust
SplunkTrust

This is not too difficult (after I figured it out) with 4 different timestamps. I handled it in my installation using the following in the props.conf file:

[tomcat_catalina]
DATETIME_CONFIG = /etc/slave-apps/all_tomcat_props/local/catalina.xml
LINE_BREAKER = ^(?:(?:\w+\s*\d{1,2},\s*\d{4}\s*\d{1,2}:\d{2}:\d{2}\s*\w+)|(?:\[\w{1,7}\]\s\d\d/\d\d/\d\d\s)|(?:\d{4}-\d{2}-\d{2}\s\d{1,2}:\d{2}:\d{2},\d{3})|(?:\[\d\d/\d\d/\d\d\s\d\d:\d\d:\d\d[:,.]\d\d\d\]))
TRUNCATE = 100000
NO_BINARY_CHECK=1
MAX_TIMESTAMP_LOOKAHEAD=200
#  Added transform to remove date from sourcename 20140324
TRANSFORMS-tcatdatestrip = tcat_strip_dates

Then I use the following catalina.xml file:

<datetime>
<define name="_mydatetimeformat4" extract="month, day, year, hour, minute, second, subsecond">
    <text>\[(\d\d)/(\d\d)/(\d\d)\s+(\d\d):(\d\d):(\d\d)[:,.](\d\d\d)\]</text>
</define>
<define name="_mydatetimeformat3" extract="year, month, day, hour, minute, second, subsecond">
     <text>(\d{4})-(\d{2})-(\d{2})\s(\d{1,2}):(\d{2}):(\d{2}).(\d{3})</text>
</define>
<define name="_mydatetimeformat2" extract="year, month, day, hour, minute, second, subsecond">
    <text>(\d\d)/(\d\d)/(\d\d)\s+(\d\d):(\d\d):(\d\d)[:,.](\d\d\d)</text>
</define>
<define name="_mydatetimeformat1" extract="litmonth, day, year, hour, minute, second, ampm">
     <text>(\w+)\s*(\d{1,2}),\s*(\d{4})\s*(\d{1,2}):(\d{2}):(\d{2})\s*(\w+)</text>
</define>
<timePatterns>
     <use name="_mydatetimeformat4"/>
     <use name="_mydatetimeformat1"/>
     <use name="_mydatetimeformat2"/>
     <use name="_mydatetimeformat3"/>
</timePatterns>
<datePatterns>
     <use name="_mydatetimeformat4"/>
     <use name="_mydatetimeformat1"/>
     <use name="_mydatetimeformat2"/>
     <use name="_mydatetimeformat3"/>
</datePatterns>
</datetime>

And for completeness, the transforms.conf file entry:

[tcat_strip_dates]
# Remove YYYY-MM-DD  style date from the filename
# /opt/TcatServer*/logs/namestuff.2014-02-13.log
#
SOURCE_KEY = MetaData:Source
DEST_KEY = MetaData:Source
REGEX = (/\w{3}/\S+/\w{4}/\w+)\.\d{4}-\d{2}-\d{2}(.log)
FORMAT = source::$1$2

If you don't have the same filenaming problem that I did (the dates in the filename), you can not to the transforms stuff in the props and transforms files.

View solution in original post

SplunkTrust
SplunkTrust

This is not too difficult (after I figured it out) with 4 different timestamps. I handled it in my installation using the following in the props.conf file:

[tomcat_catalina]
DATETIME_CONFIG = /etc/slave-apps/all_tomcat_props/local/catalina.xml
LINE_BREAKER = ^(?:(?:\w+\s*\d{1,2},\s*\d{4}\s*\d{1,2}:\d{2}:\d{2}\s*\w+)|(?:\[\w{1,7}\]\s\d\d/\d\d/\d\d\s)|(?:\d{4}-\d{2}-\d{2}\s\d{1,2}:\d{2}:\d{2},\d{3})|(?:\[\d\d/\d\d/\d\d\s\d\d:\d\d:\d\d[:,.]\d\d\d\]))
TRUNCATE = 100000
NO_BINARY_CHECK=1
MAX_TIMESTAMP_LOOKAHEAD=200
#  Added transform to remove date from sourcename 20140324
TRANSFORMS-tcatdatestrip = tcat_strip_dates

Then I use the following catalina.xml file:

<datetime>
<define name="_mydatetimeformat4" extract="month, day, year, hour, minute, second, subsecond">
    <text>\[(\d\d)/(\d\d)/(\d\d)\s+(\d\d):(\d\d):(\d\d)[:,.](\d\d\d)\]</text>
</define>
<define name="_mydatetimeformat3" extract="year, month, day, hour, minute, second, subsecond">
     <text>(\d{4})-(\d{2})-(\d{2})\s(\d{1,2}):(\d{2}):(\d{2}).(\d{3})</text>
</define>
<define name="_mydatetimeformat2" extract="year, month, day, hour, minute, second, subsecond">
    <text>(\d\d)/(\d\d)/(\d\d)\s+(\d\d):(\d\d):(\d\d)[:,.](\d\d\d)</text>
</define>
<define name="_mydatetimeformat1" extract="litmonth, day, year, hour, minute, second, ampm">
     <text>(\w+)\s*(\d{1,2}),\s*(\d{4})\s*(\d{1,2}):(\d{2}):(\d{2})\s*(\w+)</text>
</define>
<timePatterns>
     <use name="_mydatetimeformat4"/>
     <use name="_mydatetimeformat1"/>
     <use name="_mydatetimeformat2"/>
     <use name="_mydatetimeformat3"/>
</timePatterns>
<datePatterns>
     <use name="_mydatetimeformat4"/>
     <use name="_mydatetimeformat1"/>
     <use name="_mydatetimeformat2"/>
     <use name="_mydatetimeformat3"/>
</datePatterns>
</datetime>

And for completeness, the transforms.conf file entry:

[tcat_strip_dates]
# Remove YYYY-MM-DD  style date from the filename
# /opt/TcatServer*/logs/namestuff.2014-02-13.log
#
SOURCE_KEY = MetaData:Source
DEST_KEY = MetaData:Source
REGEX = (/\w{3}/\S+/\w{4}/\w+)\.\d{4}-\d{2}-\d{2}(.log)
FORMAT = source::$1$2

If you don't have the same filenaming problem that I did (the dates in the filename), you can not to the transforms stuff in the props and transforms files.

View solution in original post

SplunkTrust
SplunkTrust

You have an open `` in the xml:

<datetime>
 <!-- 2014-06-11 11:08:50 [4d313890] info    [native] dynaTrace Bootstrap Agent 5.5.0.5226 -->
 <define name="_datetimeformat1" extract="year, month, day, hour, minute, second">
 <text>(\d{4})-(\d{2})-(\d{2})\s(\d{2}):(\d{2}):(\d{2})</text>

 <?-- Right here is the extra <text>: -->
 <text>

 </define>
 <!-- 11.06.2014 11:12:47 ERROR [org.springframework.web.portlet.FrameworkPortlet.processRequest():559] Could not complete request -->
 <define name="_datetimeformat2" extract "day, month, year, hour, minute, second">
 <text>(\d{2})\.(\d{2})\.(\d{4})\s(\d{2}):(\d{2}):(\d{2})</text>
 </define>
 <timePatterns>
 <use name="_datetimeformat1"/>
 <use name="_datetimeformat2"/>
 </timePatterns>
 <datePatterns>
 <use name="_datetimeformat1"/>
 <use name="_datetimeformat2"/>
 </datePatterns>
 </datetime>

That could be your problem. Not sure about that though. Try it and let us know.

0 Karma

SplunkTrust
SplunkTrust

Looks like it didn't put the <text> in my comment previously. Where is has "You have an extra `in the xml," insert thetext` tag between the quotes.

0 Karma

Champion

This is essentially what I've done, but I get that strange error. Am I missing something?

0 Karma

Champion

I found this blog entry, and gave it whirl, but no luck.

http://www.function1.com/2013/01/oh-no-splunking-log-files-with-multiple-formats-no-problem

Here's my new datetime.xml file:

<datetime>
<!-- 2014-06-11 11:08:50 [4d313890] info    [native] dynaTrace Bootstrap Agent 5.5.0.5226 -->
<define name="_datetimeformat1" extract="year, month, day, hour, minute, second">
<text>(\d{4})-(\d{2})-(\d{2})\s(\d{2}):(\d{2}):(\d{2})</text>
<text>
</define>
<!-- 11.06.2014 11:12:47 ERROR [org.springframework.web.portlet.FrameworkPortlet.processRequest():559] Could not complete request -->
<define name="_datetimeformat2" extract "day, month, year, hour, minute, second">
<text>(\d{2})\.(\d{2})\.(\d{4})\s(\d{2}):(\d{2}):(\d{2})</text>
</define>
<timePatterns>
<use name="_datetimeformat1"/>
<use name="_datetimeformat2"/>
</timePatterns>
<datePatterns>
<use name="_datetimeformat1"/>
<use name="_datetimeformat2"/>
</datePatterns>
</datetime>

Here's my props.conf, using the data preview:
DATETIME_CONFIG = /etc/apps/SVB/datetime.xml
LINE_BREAKER LINE_BREAKER = ([\r\n]+)\d{4}\d{2}\d{2}\s\d{d}:\d{2}:\d{2([\r\n]+)\d{2}.\d{2}.\d{4}\s\d{2}:\d{2}:\d{2}
SHOULD_LINEMERGE=false

When I try it, I get:
Generated an empty preview: check your line breaking and line merging settings; Uncaught exception in Aggregator, skipping an event: Error parsing regex XML file: /apps/splunk/etc/apps/SVB/datetime.xml - Couldn't find 'timePatterns' in config data for AggregatorProcessor.

0 Karma