Getting Data In

How to index two different datestamps in a single sourcetype?

nmohammed
Builder

We had logs initially with timestamp: [05/18/15 6:00:02.3898 AM]

With the latest release, the timestamp in logs changed to [05/18/15 6:00:02.3898]. There is no local equivalent for time (AM or PM).

indexes.conf :

TIME_PREFIX = ^\[
TIME_FORMAT = %m/%d/%y %H:%M:%S.%4N %p
MAX_TIMESTAMP_LOOKAHEAD = 40
SHOULD_LINEMERGE = false
LINE_BREAKER = ([\n\r]+)\[\d+\/\d+\/\d+\s+\d+\:\d+\:\d+\.\d+\s*\w*\]
TRUNCATE = 20000
KV_MODE = none

But we wanted to index a few more old logs along with new ones with new timestamp format. What can we modify in the indexes.conf for TIME_FORMAT, or should we use any other customization? This is becoming very expensive for us while indexing, as it slows down spending time parsing on these logs. And I feel the WARN messages I see in splunkd.log are due to the above change in log time stamp formats.

07-29-2015 13:42:27.889 -0700 WARN  DateParserVerbose - Failed to parse timestamp. Defaulting to timestamp of previous event (Sun Jul 26 23:48:00 2015). 

New log examples :

[07/26/15 0:00:13.2596] ABC {067}: <67afc72a-4b5b-4e07-a458-eed2b7dc30b2>Abc.Efg("xyz") by jkin/xyz/7bb07297-ffcc-49ab-bb0e-5b9536ab3eb6. ,<0ms>

[07/26/15 0:00:13.3220] ABC {067}: <c8e592fa-df82-4247-88a0-2e819b7ca74c>Abc.Efg(77) by jking/xyz/7bb07297-ffcc-49ab-bb0e-5b9536ab3eb6. ,<406ms>

old Log examples :

[05/18/15 6:00:02.4210 AM] ABC {614}: Abc.Efg("0pk1") by 1ml3/abc/dbc910c1-0958-46d2-a497-e19c91e72408. ******** (766 MB)/35 Objects

Thanks

0 Karma

woodcock
Esteemed Legend

You can do this with using datetime.xml as indicated in this blog post:

http://www.function1.com/2013/01/oh-no-splunking-log-files-with-multiple-formats-no-problem

0 Karma
Get Updates on the Splunk Community!

See just what you’ve been missing | Observability tracks at Splunk University

Looking to sharpen your observability skills so you can better understand how to collect and analyze data from ...

Weezer at .conf25? Say it ain’t so!

Hello Splunkers, The countdown to .conf25 is on-and we've just turned up the volume! We're thrilled to ...

How SC4S Makes Suricata Logs Ingestion Simple

Network security monitoring has become increasingly critical for organizations of all sizes. Splunk has ...