Timestamp in every single line in multiline events

[can_surer]

Hi,
could you please help us about that issue.

you can see piece of log in the following lines.
thanks.

14/01/29 08:29:08 Error: will not be bootstrapped since corresponding module declaration was not found in application.xml.
14/01/30 04:01:14 Error: will not be bootstrapped since corresponding module declaration was not found in application.xml.
14/01/30 15:11:57 com.evermind.server.http.HttpIOException: Broken pipe
14/01/30 15:11:57 at com.evermind.server.http.EvermindServletOutputStream.write(EvermindServletOutputStream.java:210)
14/01/30 15:11:57 at com.evermind.server.http.EvermindJSPWriter.writeOut(EvermindJSPWriter.java:576)
14/01/30 15:11:57 at com.evermind.server.http.EvermindJSPWriter.jspflush(EvermindJSPWriter.java:441)
14/01/30 15:11:57 at com.evermind.server.http.EvermindJSPWriter.close(EvermindJSPWriter.java:411)
14/01/30 15:11:57 at oracle.jsp.runtime.OracleJspRuntime.extraHandlePCFinally(OracleJspRuntime.java:1910)
14/01/30 15:11:57 at _OA._jspService(_OA.java:260)
14/01/30 15:11:57 at com.orionserver.http.OrionHttpJspPage.service(OrionHttpJspPage.java:59)
14/01/30 15:11:57 at oracle.jsp.runtimev2.JspPageTable.service(JspPageTable.java:390)
14/01/30 15:11:57 at oracle.jsp.runtimev2.JspServlet.internalService(JspServlet.java:594)
14/01/30 15:11:57 at oracle.jsp.runtimev2.JspServlet.service(JspServlet.java:518)
14/01/30 15:11:57 at javax.servlet.http.HttpServlet.service(HttpServlet.java:856)
14/01/30 15:11:57 at com.evermind.server.http.ServletRequestDispatcher.invoke(ServletRequestDispatcher.java:734)
14/01/30 15:11:57 at com.evermind.server.http.ServletRequestDispatcher.forwardInternal(ServletRequestDispatcher.java:391)
14/01/30 15:11:57 at com.evermind.server.http.ServletRequestDispatcher.unprivileged_forward(ServletRequestDispatcher.java:280)
14/01/30 15:11:57 at com.evermind.server.http.ServletRequestDispatcher.access$100(ServletRequestDispatcher.java:68)
14/01/30 15:11:57 at com.evermind.server.http.ServletRequestDispatcher$2.oc4jRun(ServletRequestDispatcher.java:214)
14/01/30 15:11:57 at oracle.oc4j.security.OC4JSecurity.doPrivileged(OC4JSecurity.java:284)
14/01/30 15:11:57 at com.evermind.server.http.ServletRequestDispatcher.forward(ServletRequestDispatcher.java:219)
14/01/30 15:11:57 at com.evermind.server.http.EvermindPageContext.forward(EvermindPageContext.java:395)
14/01/30 15:11:57 at _RF._jspService(_RF.java:225)
14/01/30 15:11:57 at com.orionserver.http.OrionHttpJspPage.service(OrionHttpJspPage.java:59)
14/01/30 15:11:57 at oracle.jsp.runtimev2.JspPageTable.service(JspPageTable.java:390)
14/01/30 15:11:57 at oracle.jsp.runtimev2.JspServlet.internalService(JspServlet.java:594)
14/01/30 15:11:57 at oracle.jsp.runtimev2.JspServlet.service(JspServlet.java:518)
14/01/30 15:11:57 at javax.servlet.http.HttpServlet.service(HttpServlet.java:856)
14/01/30 15:11:57 at com.evermind.server.http.ResourceFilterChain.doFilter(ResourceFilterChain.java:64)
14/01/30 15:11:57 at oracle.apps.jtf.base.session.ReleaseResFilter.doFilter(ReleaseResFilter.java:26)
14/01/30 15:11:57 at com.evermind.server.http.EvermindFilterChain.doFilter(EvermindFilterChain.java:15)
14/01/30 15:11:57 at oracle.apps.fnd.security.AppsServletFilter.doFilter(AppsServletFilter.java:318)
14/01/30 15:11:57 at com.evermind.server.http.ServletRequestDispatcher.invoke(ServletRequestDispatcher.java:642)
14/01/30 15:11:57 at com.evermind.server.http.ServletRequestDispatcher.forwardInternal(ServletRequestDispatcher.java:391)
14/01/30 15:11:57 at com.evermind.server.http.HttpRequestHandler.doProcessRequest(HttpRequestHandler.java:908)
14/01/30 15:11:57 at com.evermind.server.http.HttpRequestHandler.processRequest(HttpRequestHandler.java:458)
14/01/30 15:11:57 at com.evermind.server.http.AJPRequestHandler.run(AJPRequestHandler.java:313)
14/01/30 15:11:57 at com.evermind.server.http.AJPRequestHandler.run(AJPRequestHandler.java:199)
14/01/30 15:11:57 at oracle.oc4j.network.ServerSocketReadHandler$SafeRunnable.run(ServerSocketReadHandler.java:260)
14/01/30 15:11:57 at com.evermind.util.ReleasableResourcePooledExecutor$MyWorker.run(ReleasableResourcePooledExecutor.java:303)
14/01/30 15:11:57 at java.lang.Thread.run(Thread.java:662)

[chanfoli]

Is this your log data or how splunk is indexing it? If the latter, It looks like your config is not successfully parsing the timestamp entry in these java logs or it is not set to break events on timestamps. If like most of my java logs, the event starts with a line containing a timestamp, you can normally successfully parse this by telling splunk to break events on timestamps and a combination of MAX_TIMESTAMP_LOOKAHEAD and TIME_FORMAT, I also usually specify timezone with TZ=[cont/region].

If you post the first line of an event, we may be able to suggest TIME_FORMAT strings.

If your logs are adding timestamps to every line, perhaps you could correct that on the application side. Otherwise you will want to see if the starting line uses a different timestamp format and tune splunk to only recognize that one.

[somesoni2]

Any specific patterns in event which will differentiate two events? If there any set "MUST_BREAK_AFTER" attribute in props.conf with that.