LINE_BREAKER breaks every line

[johnansett]

Hello!

I have two log's I'm battling with onboardining. The first loga.log is in the following format:

[0m02-21 07:49:08,449 ContainerBackgroundProcessor[StandardEngine[jboss.web]] [/] [ERROR]  WELD-000019 Error destroying an instance company.somethingapi.security.AuthorizationContext@15bf1401 of Managed Bean [class company.somethingapi.security.AuthorizationContext] with qualifiers [@Default @Any @Named]
02-21 07:49:12,026 EJB default - 27 [/] [WARN]  JBAS014143: A previous execution of timer [somethingServer.somethingCore.SixSecondCleanupSchedule bbc0c64e-1bd9-4568-9406-bd557aed089c] is still in progress, skipping this overlapping scheduled execution at: Thu Feb 21 07:49:12 PST 2019 as timer state is IN_TIMEOUT
02-21 07:49:18,709 ContainerBackgroundProcessor[StandardEngine[jboss.web]] [/] [ERROR]  JBAS014134: EJB Invocation failed on component UserHandlerImpl for method public company.somethingapi.bean.LoginBean company.somethingcore.handler.impl.UserHandlerImpl.logout(company.somethingapi.bean.LoginBean)
: javax.ejb.EJBException: org.apache.ibatis.exceptions.PersistenceException: 
### Error updating database.  Cause: somethingPersistenceException
    ERROR - ORA-01400: cannot insert NULL into ("AC"."something_AUDIT"."EMPLOYEE_ID")

    MAPPER -company.something.dao.mybatis.AuditMapper.insertAuditRecord
 QUERY - INSERT INTO apd_something_audit ( module_id, function_id, employee_id, timestamp_dt, machine_nm_tx, transaction_cd, key1_da, key2_da, key3_da, trans_dtl_tx ) VALUES ( SUBSTR(?,1,2), SUBSTR(?,1,10), SUBSTR(?,1,64), SYSDATE, SUBSTR(?,1,64), SUBSTR(?,1,2), SUBSTR(?,1,25), SUBSTR(?,1,25), SUBSTR(?,1,25), SUBSTR(?,1,255) )
 PARMS - [something(String)] [LOUT(String)] [NULL(UNKNOWN)] [(String)] [X(String)] [NULL(UNKNOWN)] [NULL(UNKNOWN)] [NULL(UNKNOWN)] [PC IP address is: (String)] 
### The error may involve defaultParameterMap
### The error occurred while setting parameters
### SQL: INSERT INTO apd_something_audit    (    module_id,     function_id,     employee_id,     timestamp_dt,     machine_nm_tx,     transaction_cd,     key1_da,     key2_da,     key3_da,    trans_dtl_tx   )   VALUES     (      SUBSTR(?,1,2),       SUBSTR(?,1,10),       SUBSTR(?,1,64),       SYSDATE,       SUBSTR(?,1,64),       SUBSTR(?,1,2),       SUBSTR(?,1,25),       SUBSTR(?,1,25),       SUBSTR(?,1,25),       SUBSTR(?,1,255)       )
### Cause: somethingPersistenceException
    ERROR - ORA-01400: cannot insert NULL into ("ACE_APD"."something_AUDIT"."EMPLOYEE_ID")

MORE AND MORE STUFF

02-21 07:49:18,747 ContainerBackgroundProcessor[StandardEngine[jboss.web]] [/] [ERROR]  WELD-000019 Error destroying an instance company.somethingapi.security.AuthorizationContext@16387d0 of Managed Bean [class company.somethingapi.security.AuthorizationContext] with qualifiers [@Default @Any @Named]
02-21 07:49:24,000 EJB default - 21 [/] [WARN]  JBAS014143: A previous execution of timer [somethingServer.somethingCore.SixSecondCleanupSchedule bbc0c64e-1bd9-4568-9406-bd557aed089c] is still in progress, skipping this overlapping scheduled execution at: Thu Feb 21 07:49:24 PST 2019 as timer state is IN_TIMEOUT

I'm trying to break on the [0m which is also the time prefix but for some reason it doens't work all the time. This is my sourcetype:

[ <SOURCETYPE NAME> ]
SHOULD_LINEMERGE=false
LINE_BREAKER=([\r\n]+)\[\d\m\d{2}-\d{2}
NO_BINARY_CHECK=true
CHARSET=UTF-8
MAX_TIMESTAMP_LOOKAHEAD=25
disabled=false
TIME_FORMAT=%m-%d %T,%3Q
TIME_PREFIX=[0m

If I turn SHOULD_LINEMERGE to true and LINE_BREAKER to default it works, but this is not best practice as I understand? How can I get this to work with correct LINE_BREAKER?

My next log, logb.log has events as such:

12-19 22:03:26,364 MSC service thread 1-5 [/] [WARN]  JBAS015960: Class Path entry js.jar in /dir/jboss/current/standalone/deployments/WebShared.war/WEB-INF/lib/dojo-shrinksafe-1.7.2.jar  does not point to a valid jar for a Class-Path reference.
12-19 22:03:36,367 MSC service thread 1-7 [/] [WARN]  JBAS015960: Class Path entry js.jar in /dir/jboss/current/standalone/deployments/Server.ear/lib/dojo-shrinksafe-1.7.2.jar  does not point to a valid jar for a Class-Path reference.
12-19 22:03:36,515 MSC service thread 1-7 [/] [WARN]  JBAS015960: Class Path entry xbean.jar in /dir/jboss/current/standalone/deployments/Server.ear/lib/batik-js-1.7.jar  does not point to a valid jar for a Class-Path reference.
12-19 22:03:36,655 MSC service thread 1-1 [/] [WARN]  JBAS011013: /dir/jboss/current/standalone/deployments/Server.ear/C.jar/META-INF/jboss-ejb-client.xml in subdeployment ignored. jboss-ejb-client.xml is only parsed for top level deployments.
12-19 22:03:36,787 MSC service thread 1-5 [/] [WARN]  JBAS011013: /dir/jboss/current/standalone/deployments/Server.ear/I-16.0.33.war/WEB-INF/jboss-ejb-client.xml in subdeployment ignored. jboss-ejb-client.xml is only parsed for top level deployments.
12-19 22:03:37,242 MSC service thread 1-4 [/] [WARN]  JBAS015960: Class Path entry lib/json-20090211.jar in /dir/jboss/current/standalone/deployments/Server.ear/SOME-16.0.33.war  does not point to a valid jar for a Class-Path reference.
12-19 22:03:37,244 MSC service thread 1-4 [/] [WARN]  JBAS015960: Class Path entry lib/joda-time-2.3.jar in /dir/jboss/current/standalone/deployments/Server.ear/SOME-16.0.33.war  does not point to a valid jar for a Class-Path reference.
12-19 22:03:38,778 MSC service thread 1-3 [/] [WARN]  JBAS015867: Deployment "deployment.Server.ear" is using a private module ("org.apache.httpcomponents:main") which may be changed or removed in future versions without notice.
12-19 22:03:38,782 MSC service thread 1-3 [/] [WARN]  JBAS015867: Deployment "deployment.Server.ear" is using a private module ("org.codehaus.jackson.jackson-mapper-asl:main") which may be changed or removed in future versions without notice.
12-19 22:03:38,782 MSC service thread 1-3 [/] [WARN]  JBAS015867: Deployment "deployment.Server.ear" is using a private module ("org.apache.commons.collections:main") which may be changed or removed in future versions without notice.
12-19 22:03:38,795 MSC service thread 1-3 [/] [WARN]  JBAS015867: Deployment "deployment.Server.ear" is using a private module ("org.apache.commons.beanutils:main") which may be changed or removed in future versions without notice.
2018-12-19 22:03:40,032 ERROR Could not register mbeans javax.management.InstanceAlreadyExistsException: org.apache.logging.log4j2:type=1873328760
    at com.sun.jmx.mbeanserver.Repository.addMBean(Repository.java:437)
    at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerWithRepository(DefaultMBeanServerInterceptor.java:1898)
    at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamicMBean(DefaultMBeanServerInterceptor.java:966)
    at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject(DefaultMBeanServerInterceptor.java:900)
    at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:324)
    at com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:522)
    at org.jboss.as.jmx.PluggableMBeanServerImpl$TcclMBeanServer.registerMBean(PluggableMBeanServerImpl.java:1441)
    at org.jboss.as.jmx.PluggableMBeanServerImpl.registerMBean(PluggableMBeanServerImpl.java:839)
    at org.apache.logging.log4j.core.jmx.Server.register(Server.java:375)
    at org.apache.logging.log4j.core.jmx.Server.reregisterMBeansAfterReconfigure(Server.java:167)
    at org.apache.logging.log4j.core.jmx.Server.reregisterMBeansAfterReconfigure(Server.java:143)
    at org.apache.logging.log4j.core.LoggerContext.setConfiguration(LoggerContext.java:372)
    at org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:426)
    at org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:442)
    at org.apache.logging.log4j.core.LoggerContext.start(LoggerContext.java:138)
    at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:147)
    at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:41)
    at org.apache.logging.log4j.LogManager.getContext(LogManager.java:175)
    at org.apache.logging.log4j.LogManager.getLogger(LogManager.java:426)
    at company.core.handler.impl.JanitorHandlerImpl.<clinit>(JanitorHandlerImpl.java:31)
    at java.lang.Class.forName0(Native Method)
    at java.lang.Class.forName(Class.java:348)
    at org.jboss.invocation.proxy.AbstractProxyFactory.afterClassLoad(AbstractProxyFactory.java:95)
    at org.jboss.invocation.proxy.AbstractClassFactory.defineClass(AbstractClassFactory.java:166)
    at org.jboss.invocation.proxy.AbstractProxyFactory.getCachedMethods(AbstractProxyFactory.java:150)
    at org.jboss.as.ejb3.component.stateless.StatelessComponentDescription$3.configure(StatelessComponentDescription.java:149)
    at org.jboss.as.ee.component.DefaultComponentViewConfigurator.configure(DefaultComponentViewConfigurator.java:68)
    at org.jboss.as.ee.component.deployers.EEModuleConfigurationProcessor.deploy(EEModuleConfigurationProcessor.java:91)
    at org.jboss.as.server.deployment.DeploymentUnitPhaseService.start(DeploymentUnitPhaseService.java:177)
    at org.jboss.msc.service.ServiceControllerImpl$StartTask.startService(ServiceControllerImpl.java:2064)
    at org.jboss.msc.service.ServiceControllerImpl$StartTask.run(ServiceControllerImpl.java:1987)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)

2018-12-19 22:03:40,032 ERROR Could not register mbeans javax.management.InstanceAlreadyExistsException: org.apache.logging.log4j2:type=1873328760
    at com.sun.jmx.mbeanserver.Repository.addMBean(Repository.java:437)
    at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerWithRepository(DefaultMBeanServerInterceptor.java:1898)
    at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamicMBean(DefaultMBeanServerInterceptor.java:966)
    at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject(DefaultMBeanServerInterceptor.java:900)
    at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:324)
    at com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:522)
    at org.jboss.as.jmx.PluggableMBeanServerImpl$TcclMBeanServer.registerMBean(PluggableMBeanServerImpl.java:1441)
    at org.jboss.as.jmx.PluggableMBeanServerImpl.registerMBean(PluggableMBeanServerImpl.java:839)
    at org.apache.logging.log4j.core.jmx.Server.register(Server.java:375)
    at org.apache.logging.log4j.core.jmx.Server.reregisterMBeansAfterReconfigure(Server.java:167)
    at org.apache.logging.log4j.core.jmx.Server.reregisterMBeansAfterReconfigure(Server.java:143)
    at org.apache.logging.log4j.core.LoggerContext.setConfiguration(LoggerContext.java:372)
    at org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:426)
    at org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:442)
    at org.apache.logging.log4j.core.LoggerContext.start(LoggerContext.java:138)
    at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:147)
    at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:41)
    at org.apache.logging.log4j.LogManager.getContext(LogManager.java:175)
    at org.apache.logging.log4j.LogManager.getLogger(LogManager.java:426)
    at company.core.handler.impl.JanitorHandlerImpl.<clinit>(JanitorHandlerImpl.java:31)
    at java.lang.Class.forName0(Native Method)
    at java.lang.Class.forName(Class.java:348)
    at org.jboss.invocation.proxy.AbstractProxyFactory.afterClassLoad(AbstractProxyFactory.java:95)
    at org.jboss.invocation.proxy.AbstractClassFactory.defineClass(AbstractClassFactory.java:166)
    at org.jboss.invocation.proxy.AbstractProxyFactory.getCachedMethods(AbstractProxyFactory.java:150)
    at org.jboss.as.ejb3.component.stateless.StatelessComponentDescription$3.configure(StatelessComponentDescription.java:149)
    at org.jboss.as.ee.component.DefaultComponentViewConfigurator.configure(DefaultComponentViewConfigurator.java:68)
    at org.jboss.as.ee.component.deployers.EEModuleConfigurationProcessor.deploy(EEModuleConfigurationProcessor.java:91)
    at org.jboss.as.server.deployment.DeploymentUnitPhaseService.start(DeploymentUnitPhaseService.java:177)
    at org.jboss.msc.service.ServiceControllerImpl$StartTask.startService(ServiceControllerImpl.java:2064)
    at org.jboss.msc.service.ServiceControllerImpl$StartTask.run(ServiceControllerImpl.java:1987)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)

12-19 22:03:51,719 ServerService Thread Pool -- 76 [/] [WARN]  RESTEASY000220: ClassNotFoundException: Unable to load builtin provider: org.jboss.resteasy.plugins.interceptors.encoding.AcceptEncodingGZIPFilter
12-19 22:03:52,238 ServerService Thread Pool -- 64 [/] [ERROR]  log4j:WARN No appenders could be found for logger (net.bull.javamelody).
12-19 22:03:52,238 ServerService Thread Pool -- 77 [/] [ERROR]  log4j:WARN No appenders could be found for logger (net.bull.javamelody).

I want to event break when I see [0m12-19 22:03:26 OR 12-19 22:03:26 and extract the time for each event. This is my sourcetype:

[ <SOURCETYPE NAME> ]
SHOULD_LINEMERGE=false
LINE_BREAKER=([\r\n]+)(?:\[\d\m\d{2}-\d{2}|\d{2}-\d{2}\s)
NO_BINARY_CHECK=true
CHARSET=UTF-8
MAX_TIMESTAMP_LOOKAHEAD=25
disabled=false
TIME_FORMAT=%m-%d %T,%3Q
TIME_PREFIX=[0m|^

Any help is greatly appreciated!!
Thank you

[johnansett]

Thanks for your help. Unfortunately both still seem to break incorrectly The first grabs the first couple correctly as single lines but as soon as it goes to multiline this happens:

The second log is the same, looks like it works well until it hits a multiline event, it also doesn't seem to get all the timestamps, some lines that begin at the start of the line (NOT after [0m) work while others don't.

[nickhills]

set linemerge to true

[FrankVl]

linemerge should by definition be false when using line breaker...

The sample data you posted here, contains some weird character (notepad++ shows it as ESC, here on answers is shows as . Is that also in your actual data? If so: that could explain why your regex is failing.

Try: ([\r\n]*.?)\[\d\m\d{2}-\d{2}

[nickhills]

Your first one I would change your Linebreaker and Time prefix to the below.

[ <SOURCETYPE NAME> ]
 SHOULD_LINEMERGE=false
 LINE_BREAKER=^(.+)?\dm\d{2}-\d{2}
 NO_BINARY_CHECK=true
 CHARSET=UTF-8
 MAX_TIMESTAMP_LOOKAHEAD=25
 disabled=false
 TIME_FORMAT=%m-%d %T,%3Q
 TIME_PREFIX=^(.+)?\[0m

For your second sourcetype use:

 [ <SOURCETYPE NAME> ]
 SHOULD_LINEMERGE=false
 LINE_BREAKER=(?:\[\dm\d{2}-\d{2}|\d{2}-\d{2}\s)
 NO_BINARY_CHECK=true
 CHARSET=UTF-8
 MAX_TIMESTAMP_LOOKAHEAD=25
 disabled=false
 TIME_FORMAT=%m-%d %T,%3Q
 TIME_PREFIX=^(.+)?\[0m|^