Getting Data In

Why are we getting "DateParserVerbose - Failed to parse timestamp. Defaulting to timestamp of previous event", but our logs or props.conf haven't changed?

Contributor

Splunk is showing the following WARN messages in the logs suddenly, but nothing was changed on the logs or props.conf. I feel the time spent to parse has increased, as many of our scheduled reports have slowed down and search is also affected.

07-29-2015 13:42:27.889 -0700 WARN  DateParserVerbose - Failed to parse timestamp. Defaulting to timestamp of previous event (Sun Jul 26 23:48:00 2015). Context: source::/server_logs-ep9/client/zDefaultInstance/12.123.123.123/EAppServer.ESERVER1234.2015-07-26-00.log|host::ehoat001|encompass|1549379

Log Filename - EAppServer.ESERVER1234.2015-07-26-00.log

I checked the logs but the time stamps look to be intact... not sure where we are breaking ..

Example of normal logs:

[07/27/15 13:00:14.0687] APITRACE {057}: <f0109337-1c69-45e7-a4c3-feeed7996650>OrganizationManager.GetScopedUsersWithRole(19) by user/Encompass/Plugin:GenPlugin/123-1234-5678-abcd-1238. ,<546ms>

[07/27/15 13:00:14.6147] APITRACE {057}: <2ae1cb31-649e-468d-ad89-c8e4487df206>CursorBase.GetItemCount() by user/Encompass/Plugin:Main/123-1234-5678-abcd-123. ,<0ms> Example from logs which 

Props.conf:

TIME_PREFIX = ^\[
TIME_FORMAT = %m/%d/%y %H:%M:%S.%4N %p
MAX_TIMESTAMP_LOOKAHEAD = 40
SHOULD_LINEMERGE = false
LINE_BREAKER = ([\n\r]+)\[\d+\/\d+\/\d+\s+\d+\:\d+\:\d+\.\d+\s*\w*\]
TRUNCATE = 20000
KV_MODE = none

transforms.conf

[enc_set_sourcetype_api]
REGEX = ^[^\]\n]*\]\s+APITRACE
FORMAT = sourcetype::encompass-apitrace
DEST_KEY = MetaData:Sourcetype
0 Karma

Esteemed Legend

You should use this instead (change %4N to %3N and %p to %z😞

TIME_FORMAT = %m/%d/%y %H:%M:%S.%3N %z
0 Karma

New Member

Just insert this in your props.conf sourcetype section:

DATETIME_CONFIG = CURRENT

0 Karma

Esteemed Legend

This will eliminate the error at the expense of ignoring and supplanting every event's timestamp. This is a terrible way to handle it; it will "work" but it will create other (invisible) problems.

0 Karma

Esteemed Legend

Your %p is looking for AM and PM which is not in your logs.

0 Karma

Contributor

I guess , I did , Not sure what I am missing here.??

I did have a entry in the props.conf to point the datetime.xml . And also in the datetime.xml have the Regex.
Can you help me point out the missing part?

0 Karma

Esteemed Legend

What exactly did you put in props.conf? As far as datetime.xml. I has to be in exactly the right spot on the Indexers and it should look something like this:

<datetime>
<define name="_withOrWithoutAMPM" extract="month, day, year, hour, minute, second, subsecond, ampm">
   <text>\[(\d{4})/(\d{2})/(\d{2})\s+(\d{1,2}):(\d{2}):(\d{2})\.(\d{4})\s*(.{0,2})\]</text>
</define>
<timePatterns>
   <use name="_withOrWithoutAMPM"/>
</timePatterns>
<datePatterns>
   <use name="_withOrWithoutAMPM"/>
</datePatterns>
</datetime>
0 Karma

Contributor

thanks again :

the following is my props.conf used when I pointed out to datetime.xml

[encompass]
DATE_TIME CONFIG = /etc/apps/EM_encompass_indexer/local/datetime.xml
LINE_BREAKER = ([\n\r])+(?=(\[(\d{2})/(\d{2})/(\d{2})\s\s(\d{1,2}):(\d{2}):(\d{2})\.(\d{4})\s(\w{2})\]|[(\d{2})/(\d{2})/(\d{2})\s\s(\d{1,2}):(\d{2}):(\d{2})\.(\d{4})\]))
SHOULD_LINEMERGE = false
TRUNCATE = 20000
KV_MODE = none

TRANSFORMS-enc_set_sourcetypes = enc_set_sourcetype_api, enc_set_sourcetype_err, enc_set_sourcetype_sql, enc_set_sourcetype_dbg, enc_set_sourcetype_warn, enc_set_sourcetype_info
0 Karma

Esteemed Legend

OK, so if you use this location and put my guts for datetime.xml on all of your Splunk Indexers and restart all of their Splunk instances, it should work:

$SPLUNK_HOME/etc/apps/EM_encompass_indexer/local/datetime.xml
0 Karma

Contributor

Thanks . One question would be, what should I put in props.conf for the LINE_BREAKER = ?

0 Karma

Esteemed Legend

Leave it out entirely (let it default).

0 Karma

Contributor

Tried it out. It is parsing correctly now, but It is not breaking the line at every time-stamp 😞

0 Karma

Esteemed Legend

Sorry, try this:

LINE_BREAKER = ([\n\r]+)\[\d(2)\/\d{2}\/\d{4}\s+\d+\:\d{2|\:\d{2}
0 Karma

Esteemed Legend

@nmohammed You really should come back and update your question. I assume that you got it all working so you should click Accept to close it out.

0 Karma

Contributor

cat datetime.xml

\[(\d{2})/(\d{2})/(\d{2})\s\s(\d{1,2}):(\d{2}):(\d{2})\.(\d{4})\s(\w{2})\]

\[(\d{2})/(\d{2})/(\d{2})\s\s(\d{1,2}):(\d{2}):(\d{2})\.(\d{4})\]
0 Karma

Esteemed Legend

OK, I see the problem now! Did you read the whole article? This is only 5% of what the file should contain!

0 Karma

Contributor

Thanks Woodcock,

I created a datetime.xml for my environment, but I guess it has some issues and taking effect:

\[(\d{2})/(\d{2})/(\d{2})\s\s(\d{1,2}):(\d{2}):(\d{2})\.(\d{4})\s(\w{2})\]

\[(\d{2})/(\d{2})/(\d{2})\s\s(\d{1,2}):(\d{2}):(\d{2})\.(\d{4})\]
0 Karma

Esteemed Legend

What is the complete contents of datetime.xml?

0 Karma

Contributor

Thanks , some of our client logs do have the localization for AM/PM , because of different version of product

This was not happening earlier and started just few days ago. Can we change the props.conf to index both of these formats ?

0 Karma

Esteemed Legend
0 Karma