How to index two different datestamps in a single sourcetype?

nmohammed — Wed, 29 Jul 2015 22:41:30 GMT

We had logs initially with timestamp: [05/18/15 6:00:02.3898 AM]

With the latest release, the timestamp in logs changed to [05/18/15 6:00:02.3898]. There is no local equivalent for time (AM or PM).

indexes.conf :

TIME_PREFIX = ^\[
TIME_FORMAT = %m/%d/%y %H:%M:%S.%4N %p
MAX_TIMESTAMP_LOOKAHEAD = 40
SHOULD_LINEMERGE = false
LINE_BREAKER = ([\n\r]+)\[\d+\/\d+\/\d+\s+\d+\:\d+\:\d+\.\d+\s*\w*\]
TRUNCATE = 20000
KV_MODE = none

But we wanted to index a few more old logs along with new ones with new timestamp format. What can we modify in the indexes.conf for TIME_FORMAT, or should we use any other customization? This is becoming very expensive for us while indexing, as it slows down spending time parsing on these logs. And I feel the WARN messages I see in splunkd.log are due to the above change in log time stamp formats.

07-29-2015 13:42:27.889 -0700 WARN  DateParserVerbose - Failed to parse timestamp. Defaulting to timestamp of previous event (Sun Jul 26 23:48:00 2015).

New log examples :

[07/26/15 0:00:13.2596] ABC {067}: <67afc72a-4b5b-4e07-a458-eed2b7dc30b2>Abc.Efg("xyz") by jkin/xyz/7bb07297-ffcc-49ab-bb0e-5b9536ab3eb6. ,<0ms>

[07/26/15 0:00:13.3220] ABC {067}: <c8e592fa-df82-4247-88a0-2e819b7ca74c>Abc.Efg(77) by jking/xyz/7bb07297-ffcc-49ab-bb0e-5b9536ab3eb6. ,<406ms>

old Log examples :

[05/18/15 6:00:02.4210 AM] ABC {614}: Abc.Efg("0pk1") by 1ml3/abc/dbc910c1-0958-46d2-a497-e19c91e72408. ******** (766 MB)/35 Objects

Thanks

Re: How to index two different datestamps in a single sourcetype?

woodcock — Mon, 03 Aug 2015 13:30:21 GMT

You can do this with using datetime.xml as indicated in this blog post:

http://www.function1.com/2013/01/oh-no-splunking-log-files-with-multiple-formats-no-problem

topic Re: How to index two different datestamps in a single sourcetype? in Getting Data In

How to index two different datestamps in a single sourcetype?

Re: How to index two different datestamps in a single sourcetype?