Multiple timestamps

[chiwang]

I have a log file that contains multiple formats of timestamps. Splunk, for some reasons only picks up the first one and uses it as an event boundry. Examples:

Event 1) Splunk creates one event with event time 2013-02-27 10:25:15,871.
|2013-02-27 10:25:15,871|[ACTIVE] ExecuteThread: '8' for queue: 'worker1'|com.testClass|INFO|userId|log xyz
[gcpause][Wed Feb 27 10:25:15 2013][00541] Thread "[ACTIVE] ExecuteThread: '3' for queue: 'worker1'" id=319 idx=0x468 tid=1851 was in object alloc 2199.796 ms from 340475.304 s
[gcpause][Wed Feb 27 10:25:15 2013][00541] Thread "[ACTIVE] ExecuteThread: '30' for queue: 'worker2'" id=13938 idx=0x69c tid=22637 was in object alloc 2197.764 ms from 340475.307 s
[gcpause][Wed Feb 27 10:25:15 2013][00541] Thread "[ACTIVE] ExecuteThread: '32' for queue: 'worker3'" id=15420 idx=0x6f0 tid=13669 was in object alloc 2191.594 ms from 340475.313 s


Event 2) Splunk creates one event with event time 2013-02-28 08:52:50,564.
|2013-02-28 08:52:50,564|[ACTIVE] ExecuteThread: '41' for queue: 'worker1'|com.someClass|ERROR|userId|
URI: [GET] /test/testURL
java.lang.IllegalStateException: Response already committed
stacktrace line1
stacktrace line2
<Feb 28, 2013 8:52:50 AM EST> <[ServletContext@226845581[app:test module:test.war path: spec-version:2.5]] Servlet failed with Exception
java.lang.IllegalStateException: Response already committed
stacktrace line1
stacktrace line2


Event 3) Splunk creates one event with event time 2013-02-27 11:14:05,333.
|2013-02-27 11:14:05,333|[ACTIVE] ExecuteThread: '4' for queue: 'worker'|com.testClass|INFO|userId|HttpServletRequest:
HttpServletRequest parameters:
param 1
param 2
[INFO ] [20130227 11:14:07.291] [vendorProduct] [VendorConnection] .dispatchResponses(): caught Exception during read...


props.conf:
[mysourcetype]
TZ = 'America/New_York'
NO_BINARY_CHECK = 1
pulldown_type = 1
REPORT-r13 = pipe_app_log_fields, source_metadata, source_region_metadata
MAX_EVENTS = 10000
DATETIME_CONFIG=/etc/system/local/custom_datetime.xml


I would like splunk to capture all events with timestamps in bold.
Any idea how I can get around this?

[chiwang]

I tried to get around this by providing a custom datetime.xml based on the suggestion from: http://splunk-base.splunk.com/answers/1807/2-different-timestamps-in-single-log

It worked if I manually uploaded the log file and previewed it via splunk web. The example I provided got parsed into 4 events. But once the log files got pumped in to splunk via a forwarder, multiple events are merged into one. Any other configuartion options that I should try?

[jbsplunk · Accepted solution]

So, I would suggest going through the following document, as I think your issue is probably more around event boundaries/line breaking than it is with timestamps:

http://docs.splunk.com/Documentation/Splunk/latest/Data/Indexmulti-lineevents#How_Splunk_determines_...

With regard to timestamps, here is another very useful document:

http://docs.splunk.com/Documentation/Splunk/latest/Data/Configuretimestamprecognition

It'll explain how to use TIME_PREFIX, which uses a regex to specify a pattern of what comes before the timestamp that you want to use. You'll also need to use TIME_FORMAT to specify the format of the timestamp, and MAX_TIMESTAMP_LOOKAHEAD to specify the length of the timestamp.

Between these two, I think you'll find what you need.

[chiwang]

Updated with log examples. Any idea how I can get what I want? I am trying to avoid creating different log files for each log formats.