Getting Data In

How to handle a logfile with two timestamps

a212830
Champion

Hi,

I have a catalina.out logfile that starts with one timestamp (YYYY-MM-DD HH:MM:SS), and then, after the initialization is complete, starts using a different one (MM.DD.YYYY HH:MM:SS). The customer needs all events, so I can't filter out the startup. How would this get handled?

2014-11-02 04:15:52 [cf6fb877] info    [native] dynaTrace Bootstrap Agent 5.5.0.5226
2014-11-02 04:15:52 [cf6fb877] info    [native] Using /fidcn/install/dynatrace-5.5.0 as Agent storage
2014-11-02 04:15:52 [cf6fb877] info    [native] Log file at /fidcn/install/dynatrace-5.5.0/log/dt_CTG_FidelityCentral_prod_bootstra
p_4145.log
2014-11-02 04:15:52 [cf6fb877] info    [native] Trying to connect to dtcoll-ctgprod-mmk.fmr.com:9998 for up to 60 seconds
2014-11-02 04:15:52 [cf6fb877] info    [native] Connected successfully
...
03.11.2014 06:22:45 ERROR [org.apache.jetspeed.container.JetspeedContainerServlet.doGet():350] JetspeedContainerServlet: Error rendering portlet "fcapps::FCStats": javax.portlet.PortletException: Request processing failed
javax.portlet.PortletException: Request processing failed
        at org.springframework.web.portlet.FrameworkPortlet.processRequest(FrameworkPortlet.java:544)
        at org.springframework.web.portlet.FrameworkPortlet.doDispatch(FrameworkPortlet.java:470)
        at javax.portlet.GenericPortlet.render(GenericPortlet.java:253)
        at org.apache.jetspeed.portlet.PortletObjectProxy.proxyRender(PortletObjectProxy.java:280)
        at org.apache.jetspeed.portlet.PortletObjectProxy.invoke(PortletObjectProxy.java:172)
        at $Proxy179.render(Unknown Source)
        at org.apache.jetspeed.factory.JetspeedPortletInstance.render(JetspeedPortletInstance.java:117)
        at org.apache.jetspeed.container.services.JetspeedFilterChain.doFilter(JetspeedFilterChain.java:142)
        at org.apache.jetspeed.container.services.JetspeedFilterChain.processFilter(JetspeedFilterChain.java:92)
        at org.apache.jetspeed.container.services.JetspeedFilterManager.processFilter(JetspeedFilterManager.java:112)
        at org.apache.jetspeed.container.JetspeedContainerServlet.doGet(JetspeedContainerServlet.java:290)
Tags (2)
0 Karma
1 Solution

cpetterborg
SplunkTrust
SplunkTrust

This is not too difficult (after I figured it out) with 4 different timestamps. I handled it in my installation using the following in the props.conf file:

[tomcat_catalina]
DATETIME_CONFIG = /etc/slave-apps/all_tomcat_props/local/catalina.xml
LINE_BREAKER = ^(?:(?:\w+\s*\d{1,2},\s*\d{4}\s*\d{1,2}:\d{2}:\d{2}\s*\w+)|(?:\[\w{1,7}\]\s\d\d/\d\d/\d\d\s)|(?:\d{4}-\d{2}-\d{2}\s\d{1,2}:\d{2}:\d{2},\d{3})|(?:\[\d\d/\d\d/\d\d\s\d\d:\d\d:\d\d[:,.]\d\d\d\]))
TRUNCATE = 100000
NO_BINARY_CHECK=1
MAX_TIMESTAMP_LOOKAHEAD=200
#  Added transform to remove date from sourcename 20140324
TRANSFORMS-tcatdatestrip = tcat_strip_dates

Then I use the following catalina.xml file:

<datetime>
<define name="_mydatetimeformat4" extract="month, day, year, hour, minute, second, subsecond">
    <text>\[(\d\d)/(\d\d)/(\d\d)\s+(\d\d):(\d\d):(\d\d)[:,.](\d\d\d)\]</text>
</define>
<define name="_mydatetimeformat3" extract="year, month, day, hour, minute, second, subsecond">
     <text>(\d{4})-(\d{2})-(\d{2})\s(\d{1,2}):(\d{2}):(\d{2}).(\d{3})</text>
</define>
<define name="_mydatetimeformat2" extract="year, month, day, hour, minute, second, subsecond">
    <text>(\d\d)/(\d\d)/(\d\d)\s+(\d\d):(\d\d):(\d\d)[:,.](\d\d\d)</text>
</define>
<define name="_mydatetimeformat1" extract="litmonth, day, year, hour, minute, second, ampm">
     <text>(\w+)\s*(\d{1,2}),\s*(\d{4})\s*(\d{1,2}):(\d{2}):(\d{2})\s*(\w+)</text>
</define>
<timePatterns>
     <use name="_mydatetimeformat4"/>
     <use name="_mydatetimeformat1"/>
     <use name="_mydatetimeformat2"/>
     <use name="_mydatetimeformat3"/>
</timePatterns>
<datePatterns>
     <use name="_mydatetimeformat4"/>
     <use name="_mydatetimeformat1"/>
     <use name="_mydatetimeformat2"/>
     <use name="_mydatetimeformat3"/>
</datePatterns>
</datetime>

And for completeness, the transforms.conf file entry:

[tcat_strip_dates]
# Remove YYYY-MM-DD  style date from the filename
# /opt/TcatServer*/logs/namestuff.2014-02-13.log
#
SOURCE_KEY = MetaData:Source
DEST_KEY = MetaData:Source
REGEX = (/\w{3}/\S+/\w{4}/\w+)\.\d{4}-\d{2}-\d{2}(.log)
FORMAT = source::$1$2

If you don't have the same filenaming problem that I did (the dates in the filename), you can not to the transforms stuff in the props and transforms files.

View solution in original post

cpetterborg
SplunkTrust
SplunkTrust

This is not too difficult (after I figured it out) with 4 different timestamps. I handled it in my installation using the following in the props.conf file:

[tomcat_catalina]
DATETIME_CONFIG = /etc/slave-apps/all_tomcat_props/local/catalina.xml
LINE_BREAKER = ^(?:(?:\w+\s*\d{1,2},\s*\d{4}\s*\d{1,2}:\d{2}:\d{2}\s*\w+)|(?:\[\w{1,7}\]\s\d\d/\d\d/\d\d\s)|(?:\d{4}-\d{2}-\d{2}\s\d{1,2}:\d{2}:\d{2},\d{3})|(?:\[\d\d/\d\d/\d\d\s\d\d:\d\d:\d\d[:,.]\d\d\d\]))
TRUNCATE = 100000
NO_BINARY_CHECK=1
MAX_TIMESTAMP_LOOKAHEAD=200
#  Added transform to remove date from sourcename 20140324
TRANSFORMS-tcatdatestrip = tcat_strip_dates

Then I use the following catalina.xml file:

<datetime>
<define name="_mydatetimeformat4" extract="month, day, year, hour, minute, second, subsecond">
    <text>\[(\d\d)/(\d\d)/(\d\d)\s+(\d\d):(\d\d):(\d\d)[:,.](\d\d\d)\]</text>
</define>
<define name="_mydatetimeformat3" extract="year, month, day, hour, minute, second, subsecond">
     <text>(\d{4})-(\d{2})-(\d{2})\s(\d{1,2}):(\d{2}):(\d{2}).(\d{3})</text>
</define>
<define name="_mydatetimeformat2" extract="year, month, day, hour, minute, second, subsecond">
    <text>(\d\d)/(\d\d)/(\d\d)\s+(\d\d):(\d\d):(\d\d)[:,.](\d\d\d)</text>
</define>
<define name="_mydatetimeformat1" extract="litmonth, day, year, hour, minute, second, ampm">
     <text>(\w+)\s*(\d{1,2}),\s*(\d{4})\s*(\d{1,2}):(\d{2}):(\d{2})\s*(\w+)</text>
</define>
<timePatterns>
     <use name="_mydatetimeformat4"/>
     <use name="_mydatetimeformat1"/>
     <use name="_mydatetimeformat2"/>
     <use name="_mydatetimeformat3"/>
</timePatterns>
<datePatterns>
     <use name="_mydatetimeformat4"/>
     <use name="_mydatetimeformat1"/>
     <use name="_mydatetimeformat2"/>
     <use name="_mydatetimeformat3"/>
</datePatterns>
</datetime>

And for completeness, the transforms.conf file entry:

[tcat_strip_dates]
# Remove YYYY-MM-DD  style date from the filename
# /opt/TcatServer*/logs/namestuff.2014-02-13.log
#
SOURCE_KEY = MetaData:Source
DEST_KEY = MetaData:Source
REGEX = (/\w{3}/\S+/\w{4}/\w+)\.\d{4}-\d{2}-\d{2}(.log)
FORMAT = source::$1$2

If you don't have the same filenaming problem that I did (the dates in the filename), you can not to the transforms stuff in the props and transforms files.

cpetterborg
SplunkTrust
SplunkTrust

You have an open `` in the xml:

<datetime>
 <!-- 2014-06-11 11:08:50 [4d313890] info    [native] dynaTrace Bootstrap Agent 5.5.0.5226 -->
 <define name="_datetimeformat1" extract="year, month, day, hour, minute, second">
 <text>(\d{4})-(\d{2})-(\d{2})\s(\d{2}):(\d{2}):(\d{2})</text>

 <?-- Right here is the extra <text>: -->
 <text>

 </define>
 <!-- 11.06.2014 11:12:47 ERROR [org.springframework.web.portlet.FrameworkPortlet.processRequest():559] Could not complete request -->
 <define name="_datetimeformat2" extract "day, month, year, hour, minute, second">
 <text>(\d{2})\.(\d{2})\.(\d{4})\s(\d{2}):(\d{2}):(\d{2})</text>
 </define>
 <timePatterns>
 <use name="_datetimeformat1"/>
 <use name="_datetimeformat2"/>
 </timePatterns>
 <datePatterns>
 <use name="_datetimeformat1"/>
 <use name="_datetimeformat2"/>
 </datePatterns>
 </datetime>

That could be your problem. Not sure about that though. Try it and let us know.

0 Karma

cpetterborg
SplunkTrust
SplunkTrust

Looks like it didn't put the <text> in my comment previously. Where is has "You have an extra `in the xml," insert thetext` tag between the quotes.

0 Karma

a212830
Champion

This is essentially what I've done, but I get that strange error. Am I missing something?

0 Karma

a212830
Champion

I found this blog entry, and gave it whirl, but no luck.

http://www.function1.com/2013/01/oh-no-splunking-log-files-with-multiple-formats-no-problem

Here's my new datetime.xml file:

<datetime>
<!-- 2014-06-11 11:08:50 [4d313890] info    [native] dynaTrace Bootstrap Agent 5.5.0.5226 -->
<define name="_datetimeformat1" extract="year, month, day, hour, minute, second">
<text>(\d{4})-(\d{2})-(\d{2})\s(\d{2}):(\d{2}):(\d{2})</text>
<text>
</define>
<!-- 11.06.2014 11:12:47 ERROR [org.springframework.web.portlet.FrameworkPortlet.processRequest():559] Could not complete request -->
<define name="_datetimeformat2" extract "day, month, year, hour, minute, second">
<text>(\d{2})\.(\d{2})\.(\d{4})\s(\d{2}):(\d{2}):(\d{2})</text>
</define>
<timePatterns>
<use name="_datetimeformat1"/>
<use name="_datetimeformat2"/>
</timePatterns>
<datePatterns>
<use name="_datetimeformat1"/>
<use name="_datetimeformat2"/>
</datePatterns>
</datetime>

Here's my props.conf, using the data preview:
DATETIME_CONFIG = /etc/apps/SVB/datetime.xml
LINE_BREAKER LINE_BREAKER = ([\r\n]+)\d{4}\d{2}\d{2}\s\d{d}:\d{2}:\d{2([\r\n]+)\d{2}.\d{2}.\d{4}\s\d{2}:\d{2}:\d{2}
SHOULD_LINEMERGE=false

When I try it, I get:
Generated an empty preview: check your line breaking and line merging settings; Uncaught exception in Aggregator, skipping an event: Error parsing regex XML file: /apps/splunk/etc/apps/SVB/datetime.xml - Couldn't find 'timePatterns' in config data for AggregatorProcessor.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...