Why are we getting "DateParserVerbose - Failed to parse timestamp. Defaulting to timestamp of previous event", but our logs or props.conf haven't changed?

[nmohammed]

Splunk is showing the following WARN messages in the logs suddenly, but nothing was changed on the logs or props.conf. I feel the time spent to parse has increased, as many of our scheduled reports have slowed down and search is also affected.

07-29-2015 13:42:27.889 -0700 WARN  DateParserVerbose - Failed to parse timestamp. Defaulting to timestamp of previous event (Sun Jul 26 23:48:00 2015). Context: source::/server_logs-ep9/client/zDefaultInstance/12.123.123.123/EAppServer.ESERVER1234.2015-07-26-00.log|host::ehoat001|encompass|1549379

Log Filename - EAppServer.ESERVER1234.2015-07-26-00.log

I checked the logs but the time stamps look to be intact... not sure where we are breaking ..

Example of normal logs:

[07/27/15 13:00:14.0687] APITRACE {057}: <f0109337-1c69-45e7-a4c3-feeed7996650>OrganizationManager.GetScopedUsersWithRole(19) by user/Encompass/Plugin:GenPlugin/123-1234-5678-abcd-1238. ,<546ms>

[07/27/15 13:00:14.6147] APITRACE {057}: <2ae1cb31-649e-468d-ad89-c8e4487df206>CursorBase.GetItemCount() by user/Encompass/Plugin:Main/123-1234-5678-abcd-123. ,<0ms> Example from logs which 

Props.conf:

TIME_PREFIX = ^\[
TIME_FORMAT = %m/%d/%y %H:%M:%S.%4N %p
MAX_TIMESTAMP_LOOKAHEAD = 40
SHOULD_LINEMERGE = false
LINE_BREAKER = ([\n\r]+)\[\d+\/\d+\/\d+\s+\d+\:\d+\:\d+\.\d+\s*\w*\]
TRUNCATE = 20000
KV_MODE = none

transforms.conf

[enc_set_sourcetype_api]
REGEX = ^[^\]\n]*\]\s+APITRACE
FORMAT = sourcetype::encompass-apitrace
DEST_KEY = MetaData:Sourcetype

[woodcock]

You should use this instead (change %4N to %3N and %p to %z😞

TIME_FORMAT = %m/%d/%y %H:%M:%S.%3N %z

[DavidLago]

Just insert this in your props.conf sourcetype section:

DATETIME_CONFIG = CURRENT

[woodcock]

This will eliminate the error at the expense of ignoring and supplanting every event's timestamp. This is a terrible way to handle it; it will "work" but it will create other (invisible) problems.

[woodcock]

Your %p is looking for AM and PM which is not in your logs.

[nmohammed]

I guess , I did , Not sure what I am missing here.??

I did have a entry in the props.conf to point the datetime.xml . And also in the datetime.xml have the Regex.
Can you help me point out the missing part?

[woodcock]

What exactly did you put in props.conf? As far as datetime.xml. I has to be in exactly the right spot on the Indexers and it should look something like this:

<datetime>
<define name="_withOrWithoutAMPM" extract="month, day, year, hour, minute, second, subsecond, ampm">
   <text>\[(\d{4})/(\d{2})/(\d{2})\s+(\d{1,2}):(\d{2}):(\d{2})\.(\d{4})\s*(.{0,2})\]</text>
</define>
<timePatterns>
   <use name="_withOrWithoutAMPM"/>
</timePatterns>
<datePatterns>
   <use name="_withOrWithoutAMPM"/>
</datePatterns>
</datetime>

[nmohammed]

thanks again :

the following is my props.conf used when I pointed out to datetime.xml

[encompass]
DATE_TIME CONFIG = /etc/apps/EM_encompass_indexer/local/datetime.xml
LINE_BREAKER = ([\n\r])+(?=(\[(\d{2})/(\d{2})/(\d{2})\s\s(\d{1,2}):(\d{2}):(\d{2})\.(\d{4})\s(\w{2})\]|[(\d{2})/(\d{2})/(\d{2})\s\s(\d{1,2}):(\d{2}):(\d{2})\.(\d{4})\]))
SHOULD_LINEMERGE = false
TRUNCATE = 20000
KV_MODE = none

TRANSFORMS-enc_set_sourcetypes = enc_set_sourcetype_api, enc_set_sourcetype_err, enc_set_sourcetype_sql, enc_set_sourcetype_dbg, enc_set_sourcetype_warn, enc_set_sourcetype_info

[woodcock]

OK, so if you use this location and put my guts for datetime.xml on all of your Splunk Indexers and restart all of their Splunk instances, it should work:

$SPLUNK_HOME/etc/apps/EM_encompass_indexer/local/datetime.xml

[nmohammed]

Thanks . One question would be, what should I put in props.conf for the LINE_BREAKER = ?

[woodcock]

Leave it out entirely (let it default).

[nmohammed]

Tried it out. It is parsing correctly now, but It is not breaking the line at every time-stamp 😞

[woodcock]

Sorry, try this:

LINE_BREAKER = ([\n\r]+)\[\d(2)\/\d{2}\/\d{4}\s+\d+\:\d{2|\:\d{2}

[woodcock]

@nmohammed You really should come back and update your question. I assume that you got it all working so you should click Accept to close it out.

[nmohammed]

cat datetime.xml

\[(\d{2})/(\d{2})/(\d{2})\s\s(\d{1,2}):(\d{2}):(\d{2})\.(\d{4})\s(\w{2})\]

\[(\d{2})/(\d{2})/(\d{2})\s\s(\d{1,2}):(\d{2}):(\d{2})\.(\d{4})\]

[woodcock]

OK, I see the problem now! Did you read the whole article? This is only 5% of what the file should contain!

[nmohammed]

Thanks Woodcock,

I created a datetime.xml for my environment, but I guess it has some issues and taking effect:

\[(\d{2})/(\d{2})/(\d{2})\s\s(\d{1,2}):(\d{2}):(\d{2})\.(\d{4})\s(\w{2})\]

\[(\d{2})/(\d{2})/(\d{2})\s\s(\d{1,2}):(\d{2}):(\d{2})\.(\d{4})\]

[woodcock]

What is the complete contents of datetime.xml?

[nmohammed]

Thanks , some of our client logs do have the localization for AM/PM , because of different version of product

This was not happening earlier and started just few days ago. Can we change the props.conf to index both of these formats ?

[woodcock]

Yes, like this:

http://www.function1.com/2013/01/oh-no-splunking-log-files-with-multiple-formats-no-problem