Why are some events being indexed with the wrong timestamp?

[alexanderadler]

Hi! I have a big Splunk enterprise environment, but I'm experiencing a strange issue where some events are losing part of their timestamp. These are the timestamps for the events

2019-01-08 07:05:32,776 StatisticMessage ,
2019-01-08 07:05:33,166 StatisticMessage , 
2019-01-08 07:05:33,401 StatisticMessage , 

and this is the props.conf file that is deployed.

[NameOfSourceType]
MAX_TIMESTAMP_LOOKAHEAD = 60
TIME_FORMAT = %Y-%m-%d %H:%M:%S,%3N
SHOULD_LINEMERGE = false
LINE_BREAKER = ([\n\r]+)\d{4}\-\d{2}\-\d{2}\s\d{2}\:\d{2}\:\d{2}\,\d{3}
TRUNCATE = 1999999

But some events are getting part of their time scrambled as shown below.

[richgalloway]

Have you seen any DateParserVerbose warnings in splunkd.log about this source/sourcetype?

[lakshman239]

I assume you want to extract the date and time from the first 24 chars, so try the below:
//limiting the lookahead, adding prefix and changing line breaker to be positive lookahead

[NameOfSourceType]
MAX_TIMESTAMP_LOOKAHEAD = 24
TIME_PREFIX = ^
TIME_FORMAT = %Y-%m-%d %H:%M:%S,%3N
SHOULD_LINEMERGE = false
LINE_BREAKER = ([\n\r]+)(?=\d{4}-\d{2}-\d{2}\s\d{2}:\d{2}:\d{2}\,\d{3}\s)
TRUNCATE = 999999

[alexanderadler]

Hi! Tried this but still the same outcome.

[lakshman239]

send sample file with a few events [ pls remove/mask sensitive data]