I have some Tomcat Catalina data and I can't for the life of me figure out why it isn't line breaking properly. There are several different formats for the data going into the catalina.out
files, so I have a rather complicated way to have to do the props.conf
file. Here is a sample of the data that made it into a single event, but it needs to be broken up into individual events for each line (in this format):
12:50:59.524 [pool-4-thread-8046] DEBUG o.s.jdbc.core.JdbcTemplate - Executing SQL query [SELECT name, MAX(version) AS version FROM (SELECT sm.name, version, direction, crt_ts, ROW_NUMBER () OVER (PARTITION BY name, version ORDER BY crt_ts DESC) AS rn FROM schema_migrations sm) WHERE rn = 1 AND direction = 'FORWARD' GROUP BY name]
12:50:59.524 [pool-4-thread-8046] DEBUG o.s.jdbc.datasource.DataSourceUtils - Fetching JDBC Connection from DataSource
12:50:59.527 [pool-4-thread-8046] DEBUG o.s.jdbc.core.BeanPropertyRowMapper - Mapping column 'NAME' to property 'name' of type class java.lang.String
12:50:59.528 [pool-4-thread-8046] DEBUG o.s.jdbc.core.BeanPropertyRowMapper - Mapping column 'VERSION' to property 'version' of type class java.lang.Integer
12:50:59.528 [pool-4-thread-8046] DEBUG o.s.jdbc.datasource.DataSourceUtils - Returning JDBC Connection to DataSource
So there is no date, only a time in this case. The props.conf
looks like this:
[tomcat_catalina]
SHOULD_LINEMERGE=true
NO_BINARY_CHECK=true
LINE_BREAKER=^(?:(?:\d{1,2}:\d\d:\d\d.\d\d\d\s+\[pool-)|(?:\[\w{1,7}\]\s\d\d\/\d\d\/\d\d\s)|(\w{3}\s+\d\d,\s\d{4}\s\d{1,2}:\d\d:\d\d\s[AP]M))
MAX_TIMESTAMP_LOOKAHEAD=200
TRUNCATE=999999
DATETIME_CONFIG = /etc/slave-apps/all_tomcat_props/local/datetime.xml
TIME_PREFIX = ^(\[\w{1,7}\]\s)?
The datetime.xml
file looks like this:
<datetime>
<define name="_mydatetimeformat3" extract="hour, minute, second, subsecond">
<text>(\d{1,2}):(\d{2}):(\d{2}).(\d{3})</text>
</define>
<define name="_mydatetimeformat2" extract="year, month, day, hour, minute, second, subsecond">
<text>(\d\d)/(\d\d)/(\d\d)\s+(\d\d):(\d\d):(\d\d)[:,.](\d\d\d)</text>
</define>
<define name="_mydatetimeformat1" extract="litmonth, day, year, hour, minute, second, ampm">
<text>(\w{3})\s+(\d{1,2}),\s+(\d{4})\s+(\d{1,2}):(\d{2}):(\d{2})\s+(\w{2})</text>
</define>
<timePatterns>
<use name="_mydatetimeformat3"/>
<use name="_mydatetimeformat2"/>
<use name="_mydatetimeformat1"/>
</timePatterns>
<datePatterns>
<use name="_mydatetimeformat3"/>
<use name="_mydatetimeformat2"/>
<use name="_mydatetimeformat1"/>
</datePatterns>
</datetime>
The LINE_BREAKER
config parameter should be getting the event to break. The first option in the pattern is the one that matches the data example, and it works fine in regex101.com
, so I don't think that is the problem. The other formats work just fine. The timestamp pattern ( _mydatetimeformat3
) is getting the timestamp parsed properly, too.
So how do I get the event breaking to work properly in this case
Remember that LINE_BREAKER throws away the matching capture group so don't include in the group anything you need in the event (like time).
Remember that LINE_BREAKER throws away the matching capture group so don't include in the group anything you need in the event (like time).
I changed it to BREAK_ONLY_BEFORE and it worked. Stupid mistake. I forgot about that little detail.
Thanks Rich!!!