topic Re: Why are we getting "DateParserVerbose - Failed to parse timestamp. Defaulting to timestamp of previous event", but our logs or props.conf haven't changed? in Getting Data In

Why are we getting "DateParserVerbose - Failed to parse timestamp. Defaulting to timestamp of previous event", but our logs or props.conf haven't changed?

nmohammed — Thu, 06 Aug 2015 20:34:45 GMT

Splunk is showing the following WARN messages in the logs suddenly, but nothing was changed on the logs or props.conf. I feel the time spent to parse has increased, as many of our scheduled reports have slowed down and search is also affected.

07-29-2015 13:42:27.889 -0700 WARN  DateParserVerbose - Failed to parse timestamp. Defaulting to timestamp of previous event (Sun Jul 26 23:48:00 2015). Context: source::/server_logs-ep9/client/zDefaultInstance/12.123.123.123/EAppServer.ESERVER1234.2015-07-26-00.log|host::ehoat001|encompass|1549379

Log Filename - EAppServer.ESERVER1234.2015-07-26-00.log

I checked the logs but the time stamps look to be intact... not sure where we are breaking ..

Example of normal logs:

[07/27/15 13:00:14.0687] APITRACE {057}: <f0109337-1c69-45e7-a4c3-feeed7996650>OrganizationManager.GetScopedUsersWithRole(19) by user/Encompass/Plugin:GenPlugin/123-1234-5678-abcd-1238. ,<546ms>

[07/27/15 13:00:14.6147] APITRACE {057}: <2ae1cb31-649e-468d-ad89-c8e4487df206>CursorBase.GetItemCount() by user/Encompass/Plugin:Main/123-1234-5678-abcd-123. ,<0ms> Example from logs which

Props.conf:

TIME_PREFIX = ^\[
TIME_FORMAT = %m/%d/%y %H:%M:%S.%4N %p
MAX_TIMESTAMP_LOOKAHEAD = 40
SHOULD_LINEMERGE = false
LINE_BREAKER = ([\n\r]+)\[\d+\/\d+\/\d+\s+\d+\:\d+\:\d+\.\d+\s*\w*\]
TRUNCATE = 20000
KV_MODE = none

transforms.conf

[enc_set_sourcetype_api]
REGEX = ^[^\]\n]*\]\s+APITRACE
FORMAT = sourcetype::encompass-apitrace
DEST_KEY = MetaData:Sourcetype

Re: Why are we getting "DateParserVerbose - Failed to parse timestamp. Defaulting to timestamp of previous event", but our logs or props.conf haven't changed?

woodcock — Thu, 06 Aug 2015 21:12:56 GMT

Your %p is looking for AM and PM which is not in your logs.

Re: Why are we getting "DateParserVerbose - Failed to parse timestamp. Defaulting to timestamp of previous event", but our logs or props.conf haven't changed?

nmohammed — Thu, 06 Aug 2015 21:17:36 GMT

Thanks , some of our client logs do have the localization for AM/PM , because of different version of product

This was not happening earlier and started just few days ago. Can we change the props.conf to index both of these formats ?

Re: Why are we getting "DateParserVerbose - Failed to parse timestamp. Defaulting to timestamp of previous event", but our logs or props.conf haven't changed?

woodcock — Thu, 06 Aug 2015 21:33:29 GMT

Yes, like this:

http://www.function1.com/2013/01/oh-no-splunking-log-files-with-multiple-formats-no-problem

Re: Why are we getting "DateParserVerbose - Failed to parse timestamp. Defaulting to timestamp of previous event", but our logs or props.conf haven't changed?

nmohammed — Mon, 10 Aug 2015 16:39:46 GMT

Thanks Woodcock,

I created a datetime.xml for my environment, but I guess it has some issues and taking effect:

\[(\d{2})/(\d{2})/(\d{2})\s\s(\d{1,2}):(\d{2}):(\d{2})\.(\d{4})\s(\w{2})\]

\[(\d{2})/(\d{2})/(\d{2})\s\s(\d{1,2}):(\d{2}):(\d{2})\.(\d{4})\]

Re: Why are we getting "DateParserVerbose - Failed to parse timestamp. Defaulting to timestamp of previous event", but our logs or props.conf haven't changed?

woodcock — Mon, 10 Aug 2015 18:23:30 GMT

What is the complete contents of datetime.xml?

Re: Why are we getting "DateParserVerbose - Failed to parse timestamp. Defaulting to timestamp of previous event", but our logs or props.conf haven't changed?

nmohammed — Mon, 10 Aug 2015 18:27:59 GMT

cat datetime.xml

\[(\d{2})/(\d{2})/(\d{2})\s\s(\d{1,2}):(\d{2}):(\d{2})\.(\d{4})\s(\w{2})\]

\[(\d{2})/(\d{2})/(\d{2})\s\s(\d{1,2}):(\d{2}):(\d{2})\.(\d{4})\]

Re: Why are we getting "DateParserVerbose - Failed to parse timestamp. Defaulting to timestamp of previous event", but our logs or props.conf haven't changed?

woodcock — Mon, 10 Aug 2015 18:36:25 GMT

OK, I see the problem now! Did you read the whole article? This is only 5% of what the file should contain!

Re: Why are we getting "DateParserVerbose - Failed to parse timestamp. Defaulting to timestamp of previous event", but our logs or props.conf haven't changed?

nmohammed — Mon, 10 Aug 2015 18:43:15 GMT

I guess , I did , Not sure what I am missing here.??

I did have a entry in the props.conf to point the datetime.xml . And also in the datetime.xml have the Regex.
Can you help me point out the missing part?

Re: Why are we getting "DateParserVerbose - Failed to parse timestamp. Defaulting to timestamp of previous event", but our logs or props.conf haven't changed?

woodcock — Mon, 10 Aug 2015 21:33:26 GMT

What exactly did you put in props.conf? As far as datetime.xml. I has to be in exactly the right spot on the Indexers and it should look something like this:

<datetime>
<define name="_withOrWithoutAMPM" extract="month, day, year, hour, minute, second, subsecond, ampm">
   <text>\[(\d{4})/(\d{2})/(\d{2})\s+(\d{1,2}):(\d{2}):(\d{2})\.(\d{4})\s*(.{0,2})\]</text>
</define>
<timePatterns>
   <use name="_withOrWithoutAMPM"/>
</timePatterns>
<datePatterns>
   <use name="_withOrWithoutAMPM"/>
</datePatterns>
</datetime>

Re: Why are we getting "DateParserVerbose - Failed to parse timestamp. Defaulting to timestamp of previous event", but our logs or props.conf haven't changed?

nmohammed — Mon, 10 Aug 2015 22:27:45 GMT

thanks again :

the following is my props.conf used when I pointed out to datetime.xml

[encompass]
DATE_TIME CONFIG = /etc/apps/EM_encompass_indexer/local/datetime.xml
LINE_BREAKER = ([\n\r])+(?=(\[(\d{2})/(\d{2})/(\d{2})\s\s(\d{1,2}):(\d{2}):(\d{2})\.(\d{4})\s(\w{2})\]|[(\d{2})/(\d{2})/(\d{2})\s\s(\d{1,2}):(\d{2}):(\d{2})\.(\d{4})\]))
SHOULD_LINEMERGE = false
TRUNCATE = 20000
KV_MODE = none

TRANSFORMS-enc_set_sourcetypes = enc_set_sourcetype_api, enc_set_sourcetype_err, enc_set_sourcetype_sql, enc_set_sourcetype_dbg, enc_set_sourcetype_warn, enc_set_sourcetype_info

Re: Why are we getting "DateParserVerbose - Failed to parse timestamp. Defaulting to timestamp of previous event", but our logs or props.conf haven't changed?

woodcock — Sat, 15 Aug 2015 21:06:34 GMT

OK, so if you use this location and put my guts for datetime.xml on all of your Splunk Indexers and restart all of their Splunk instances, it should work:

$SPLUNK_HOME/etc/apps/EM_encompass_indexer/local/datetime.xml

Re: Why are we getting "DateParserVerbose - Failed to parse timestamp. Defaulting to timestamp of previous event", but our logs or props.conf haven't changed?

nmohammed — Sun, 16 Aug 2015 00:59:16 GMT

Thanks . One question would be, what should I put in props.conf for the LINE_BREAKER = ?

Re: Why are we getting "DateParserVerbose - Failed to parse timestamp. Defaulting to timestamp of previous event", but our logs or props.conf haven't changed?

woodcock — Sun, 16 Aug 2015 19:01:18 GMT

Leave it out entirely (let it default).

Re: Why are we getting "DateParserVerbose - Failed to parse timestamp. Defaulting to timestamp of previous event", but our logs or props.conf haven't changed?

nmohammed — Mon, 17 Aug 2015 17:56:12 GMT

Tried it out. It is parsing correctly now, but It is not breaking the line at every time-stamp 😞

Re: Why are we getting "DateParserVerbose - Failed to parse timestamp. Defaulting to timestamp of previous event", but our logs or props.conf haven't changed?

woodcock — Mon, 17 Aug 2015 22:10:49 GMT

Sorry, try this:

LINE_BREAKER = ([\n\r]+)\[\d(2)\/\d{2}\/\d{4}\s+\d+\:\d{2|\:\d{2}

Re: Why are we getting "DateParserVerbose - Failed to parse timestamp. Defaulting to timestamp of previous event", but our logs or props.conf haven't changed?

DavidLago — Wed, 03 May 2017 16:37:12 GMT

Just insert this in your props.conf sourcetype section:

DATETIME_CONFIG = CURRENT

Re: Why are we getting "DateParserVerbose - Failed to parse timestamp. Defaulting to timestamp of previous event", but our logs or props.conf haven't changed?

woodcock — Wed, 03 May 2017 19:46:37 GMT

You should use this instead (change %4N to %3N and %p to %z😞

TIME_FORMAT = %m/%d/%y %H:%M:%S.%3N %z

Re: Why are we getting "DateParserVerbose - Failed to parse timestamp. Defaulting to timestamp of previous event", but our logs or props.conf haven't changed?

woodcock — Wed, 03 May 2017 19:48:44 GMT

This will eliminate the error at the expense of ignoring and supplanting every event's timestamp. This is a terrible way to handle it; it will "work" but it will create other (invisible) problems.

Re: Why are we getting "DateParserVerbose - Failed to parse timestamp. Defaulting to timestamp of previous event", but our logs or props.conf haven't changed?

woodcock — Sat, 12 Oct 2019 16:46:41 GMT

@nmohammed You really should come back and update your question. I assume that you got it all working so you should click Accept to close it out.