Getting Data In

How to handle a logfile with two timestamps

a212830
Champion

Hi,

I have a catalina.out logfile that starts with one timestamp (YYYY-MM-DD HH:MM:SS), and then, after the initialization is complete, starts using a different one (MM.DD.YYYY HH:MM:SS). The customer needs all events, so I can't filter out the startup. How would this get handled?

2014-11-02 04:15:52 [cf6fb877] info    [native] dynaTrace Bootstrap Agent 5.5.0.5226
2014-11-02 04:15:52 [cf6fb877] info    [native] Using /fidcn/install/dynatrace-5.5.0 as Agent storage
2014-11-02 04:15:52 [cf6fb877] info    [native] Log file at /fidcn/install/dynatrace-5.5.0/log/dt_CTG_FidelityCentral_prod_bootstra
p_4145.log
2014-11-02 04:15:52 [cf6fb877] info    [native] Trying to connect to dtcoll-ctgprod-mmk.fmr.com:9998 for up to 60 seconds
2014-11-02 04:15:52 [cf6fb877] info    [native] Connected successfully
...
03.11.2014 06:22:45 ERROR [org.apache.jetspeed.container.JetspeedContainerServlet.doGet():350] JetspeedContainerServlet: Error rendering portlet "fcapps::FCStats": javax.portlet.PortletException: Request processing failed
javax.portlet.PortletException: Request processing failed
        at org.springframework.web.portlet.FrameworkPortlet.processRequest(FrameworkPortlet.java:544)
        at org.springframework.web.portlet.FrameworkPortlet.doDispatch(FrameworkPortlet.java:470)
        at javax.portlet.GenericPortlet.render(GenericPortlet.java:253)
        at org.apache.jetspeed.portlet.PortletObjectProxy.proxyRender(PortletObjectProxy.java:280)
        at org.apache.jetspeed.portlet.PortletObjectProxy.invoke(PortletObjectProxy.java:172)
        at $Proxy179.render(Unknown Source)
        at org.apache.jetspeed.factory.JetspeedPortletInstance.render(JetspeedPortletInstance.java:117)
        at org.apache.jetspeed.container.services.JetspeedFilterChain.doFilter(JetspeedFilterChain.java:142)
        at org.apache.jetspeed.container.services.JetspeedFilterChain.processFilter(JetspeedFilterChain.java:92)
        at org.apache.jetspeed.container.services.JetspeedFilterManager.processFilter(JetspeedFilterManager.java:112)
        at org.apache.jetspeed.container.JetspeedContainerServlet.doGet(JetspeedContainerServlet.java:290)
Tags (2)
0 Karma
1 Solution

cpetterborg
SplunkTrust
SplunkTrust

This is not too difficult (after I figured it out) with 4 different timestamps. I handled it in my installation using the following in the props.conf file:

[tomcat_catalina]
DATETIME_CONFIG = /etc/slave-apps/all_tomcat_props/local/catalina.xml
LINE_BREAKER = ^(?:(?:\w+\s*\d{1,2},\s*\d{4}\s*\d{1,2}:\d{2}:\d{2}\s*\w+)|(?:\[\w{1,7}\]\s\d\d/\d\d/\d\d\s)|(?:\d{4}-\d{2}-\d{2}\s\d{1,2}:\d{2}:\d{2},\d{3})|(?:\[\d\d/\d\d/\d\d\s\d\d:\d\d:\d\d[:,.]\d\d\d\]))
TRUNCATE = 100000
NO_BINARY_CHECK=1
MAX_TIMESTAMP_LOOKAHEAD=200
#  Added transform to remove date from sourcename 20140324
TRANSFORMS-tcatdatestrip = tcat_strip_dates

Then I use the following catalina.xml file:

<datetime>
<define name="_mydatetimeformat4" extract="month, day, year, hour, minute, second, subsecond">
    <text>\[(\d\d)/(\d\d)/(\d\d)\s+(\d\d):(\d\d):(\d\d)[:,.](\d\d\d)\]</text>
</define>
<define name="_mydatetimeformat3" extract="year, month, day, hour, minute, second, subsecond">
     <text>(\d{4})-(\d{2})-(\d{2})\s(\d{1,2}):(\d{2}):(\d{2}).(\d{3})</text>
</define>
<define name="_mydatetimeformat2" extract="year, month, day, hour, minute, second, subsecond">
    <text>(\d\d)/(\d\d)/(\d\d)\s+(\d\d):(\d\d):(\d\d)[:,.](\d\d\d)</text>
</define>
<define name="_mydatetimeformat1" extract="litmonth, day, year, hour, minute, second, ampm">
     <text>(\w+)\s*(\d{1,2}),\s*(\d{4})\s*(\d{1,2}):(\d{2}):(\d{2})\s*(\w+)</text>
</define>
<timePatterns>
     <use name="_mydatetimeformat4"/>
     <use name="_mydatetimeformat1"/>
     <use name="_mydatetimeformat2"/>
     <use name="_mydatetimeformat3"/>
</timePatterns>
<datePatterns>
     <use name="_mydatetimeformat4"/>
     <use name="_mydatetimeformat1"/>
     <use name="_mydatetimeformat2"/>
     <use name="_mydatetimeformat3"/>
</datePatterns>
</datetime>

And for completeness, the transforms.conf file entry:

[tcat_strip_dates]
# Remove YYYY-MM-DD  style date from the filename
# /opt/TcatServer*/logs/namestuff.2014-02-13.log
#
SOURCE_KEY = MetaData:Source
DEST_KEY = MetaData:Source
REGEX = (/\w{3}/\S+/\w{4}/\w+)\.\d{4}-\d{2}-\d{2}(.log)
FORMAT = source::$1$2

If you don't have the same filenaming problem that I did (the dates in the filename), you can not to the transforms stuff in the props and transforms files.

View solution in original post

cpetterborg
SplunkTrust
SplunkTrust

This is not too difficult (after I figured it out) with 4 different timestamps. I handled it in my installation using the following in the props.conf file:

[tomcat_catalina]
DATETIME_CONFIG = /etc/slave-apps/all_tomcat_props/local/catalina.xml
LINE_BREAKER = ^(?:(?:\w+\s*\d{1,2},\s*\d{4}\s*\d{1,2}:\d{2}:\d{2}\s*\w+)|(?:\[\w{1,7}\]\s\d\d/\d\d/\d\d\s)|(?:\d{4}-\d{2}-\d{2}\s\d{1,2}:\d{2}:\d{2},\d{3})|(?:\[\d\d/\d\d/\d\d\s\d\d:\d\d:\d\d[:,.]\d\d\d\]))
TRUNCATE = 100000
NO_BINARY_CHECK=1
MAX_TIMESTAMP_LOOKAHEAD=200
#  Added transform to remove date from sourcename 20140324
TRANSFORMS-tcatdatestrip = tcat_strip_dates

Then I use the following catalina.xml file:

<datetime>
<define name="_mydatetimeformat4" extract="month, day, year, hour, minute, second, subsecond">
    <text>\[(\d\d)/(\d\d)/(\d\d)\s+(\d\d):(\d\d):(\d\d)[:,.](\d\d\d)\]</text>
</define>
<define name="_mydatetimeformat3" extract="year, month, day, hour, minute, second, subsecond">
     <text>(\d{4})-(\d{2})-(\d{2})\s(\d{1,2}):(\d{2}):(\d{2}).(\d{3})</text>
</define>
<define name="_mydatetimeformat2" extract="year, month, day, hour, minute, second, subsecond">
    <text>(\d\d)/(\d\d)/(\d\d)\s+(\d\d):(\d\d):(\d\d)[:,.](\d\d\d)</text>
</define>
<define name="_mydatetimeformat1" extract="litmonth, day, year, hour, minute, second, ampm">
     <text>(\w+)\s*(\d{1,2}),\s*(\d{4})\s*(\d{1,2}):(\d{2}):(\d{2})\s*(\w+)</text>
</define>
<timePatterns>
     <use name="_mydatetimeformat4"/>
     <use name="_mydatetimeformat1"/>
     <use name="_mydatetimeformat2"/>
     <use name="_mydatetimeformat3"/>
</timePatterns>
<datePatterns>
     <use name="_mydatetimeformat4"/>
     <use name="_mydatetimeformat1"/>
     <use name="_mydatetimeformat2"/>
     <use name="_mydatetimeformat3"/>
</datePatterns>
</datetime>

And for completeness, the transforms.conf file entry:

[tcat_strip_dates]
# Remove YYYY-MM-DD  style date from the filename
# /opt/TcatServer*/logs/namestuff.2014-02-13.log
#
SOURCE_KEY = MetaData:Source
DEST_KEY = MetaData:Source
REGEX = (/\w{3}/\S+/\w{4}/\w+)\.\d{4}-\d{2}-\d{2}(.log)
FORMAT = source::$1$2

If you don't have the same filenaming problem that I did (the dates in the filename), you can not to the transforms stuff in the props and transforms files.

cpetterborg
SplunkTrust
SplunkTrust

You have an open `` in the xml:

<datetime>
 <!-- 2014-06-11 11:08:50 [4d313890] info    [native] dynaTrace Bootstrap Agent 5.5.0.5226 -->
 <define name="_datetimeformat1" extract="year, month, day, hour, minute, second">
 <text>(\d{4})-(\d{2})-(\d{2})\s(\d{2}):(\d{2}):(\d{2})</text>

 <?-- Right here is the extra <text>: -->
 <text>

 </define>
 <!-- 11.06.2014 11:12:47 ERROR [org.springframework.web.portlet.FrameworkPortlet.processRequest():559] Could not complete request -->
 <define name="_datetimeformat2" extract "day, month, year, hour, minute, second">
 <text>(\d{2})\.(\d{2})\.(\d{4})\s(\d{2}):(\d{2}):(\d{2})</text>
 </define>
 <timePatterns>
 <use name="_datetimeformat1"/>
 <use name="_datetimeformat2"/>
 </timePatterns>
 <datePatterns>
 <use name="_datetimeformat1"/>
 <use name="_datetimeformat2"/>
 </datePatterns>
 </datetime>

That could be your problem. Not sure about that though. Try it and let us know.

0 Karma

cpetterborg
SplunkTrust
SplunkTrust

Looks like it didn't put the <text> in my comment previously. Where is has "You have an extra `in the xml," insert thetext` tag between the quotes.

0 Karma

a212830
Champion

This is essentially what I've done, but I get that strange error. Am I missing something?

0 Karma

a212830
Champion

I found this blog entry, and gave it whirl, but no luck.

http://www.function1.com/2013/01/oh-no-splunking-log-files-with-multiple-formats-no-problem

Here's my new datetime.xml file:

<datetime>
<!-- 2014-06-11 11:08:50 [4d313890] info    [native] dynaTrace Bootstrap Agent 5.5.0.5226 -->
<define name="_datetimeformat1" extract="year, month, day, hour, minute, second">
<text>(\d{4})-(\d{2})-(\d{2})\s(\d{2}):(\d{2}):(\d{2})</text>
<text>
</define>
<!-- 11.06.2014 11:12:47 ERROR [org.springframework.web.portlet.FrameworkPortlet.processRequest():559] Could not complete request -->
<define name="_datetimeformat2" extract "day, month, year, hour, minute, second">
<text>(\d{2})\.(\d{2})\.(\d{4})\s(\d{2}):(\d{2}):(\d{2})</text>
</define>
<timePatterns>
<use name="_datetimeformat1"/>
<use name="_datetimeformat2"/>
</timePatterns>
<datePatterns>
<use name="_datetimeformat1"/>
<use name="_datetimeformat2"/>
</datePatterns>
</datetime>

Here's my props.conf, using the data preview:
DATETIME_CONFIG = /etc/apps/SVB/datetime.xml
LINE_BREAKER LINE_BREAKER = ([\r\n]+)\d{4}\d{2}\d{2}\s\d{d}:\d{2}:\d{2([\r\n]+)\d{2}.\d{2}.\d{4}\s\d{2}:\d{2}:\d{2}
SHOULD_LINEMERGE=false

When I try it, I get:
Generated an empty preview: check your line breaking and line merging settings; Uncaught exception in Aggregator, skipping an event: Error parsing regex XML file: /apps/splunk/etc/apps/SVB/datetime.xml - Couldn't find 'timePatterns' in config data for AggregatorProcessor.

0 Karma
Get Updates on the Splunk Community!

What You Read The Most: Splunk Lantern’s Most Popular Articles!

Splunk Lantern is a Splunk customer success center that provides advice from Splunk experts on valuable data ...

See your relevant APM services, dashboards, and alerts in one place with the updated ...

As a Splunk Observability user, you have a lot of data you have to manage, prioritize, and troubleshoot on a ...

Index This | What goes away as soon as you talk about it?

May 2025 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with this month’s ...