Incorrect Timestamp carried from previous events

[nareshinsvu]

Hello Experts,

I am indexing data from a shared file. I have below config in my props.conf. Some of the lines from my inout log file doesn't have timestamp. So, All those events are getting timestamp from previous events read by Splunk.

Using this config, I am getting irregular timestamps captured. Any advice to fix this is much appreciated.

[Custom_W22]
DATETIME_CONFIG =
NO_BINARY_CHECK = true
SHOULD_LINEMERGE = false
TIME_FORMAT = %Y-%m-%d_%H:%M:%S.%3N
TIME_PREFIX = ^
TRANSFORMS-set = discardAll,queue2resp,index2resp
category = Custom
disabled = false
pulldown_type = 1

Example from log:
2019-02-20_03:30:02.333 - Line-1
2019-02-20_03:30:02.349 - Line-2
2019-02-20_03:30:02.364 - Line-3
2019-02-20_03:30:02.380 - Line-4
- Line-5
2019-02-20_03:30:02.427 - Line-6

Expected Output: Line-5 should have the timestamp of either Line-6 or Line-4. But it is going out of these bounds and showing different timestamps for few lines. Any help please?

[nickhills · Accepted solution]

Splunk makes an assumption (which is generally sensible) that all log lines from the same file, are from the same sourcetype.

Your props.conf example, defines a sourcetype which has the configuration also listed above.
Splunk therefore expects every event in that file to follow a standard format - in your case this says that an event ALWAYS starts with a date (from your TIME_PREFIX=^ configuration)

You also have SHOULD_LINEMERGE = false which means treat every line as a seperate event, but since Line 5 has no date, it cant be matched correctly, and probably instead using the index time.

You may want to consider if you should 'merge' these lines, or use a different line breaking process (see: https://docs.splunk.com/Documentation/Splunk/7.2.4/Data/Configureeventlinebreaking)

or
Get these events into a different sourcetype by re-writing the sourcetype as you index them: https://docs.splunk.com/Documentation/Splunk/7.2.4/Data/Advancedsourcetypeoverrides

or
by getting them into a different source file (preferable IMHO)

[nareshinsvu]

It worked when I redirected to a different sourcetype turning on the SHOULD_LINEMERGE . Thanks