Why are events with equal timestamps merged into one event?

[zapping575]

I have a few files in which the log events happen to not be in chronological order.

Specifically, an event with say, timestamp "2022-01-01 11:00:00" may occur towards the top of the log, while a different event (with a different event message) with the same timestamp may occur towards the bottom of the log.

It is totally acceptable to have log events where the timestamps are exactly equal.

What splunk is doing however, is merging all of these "distributed" events together into one single event. This should not happen.

These are my config files:

props.conf
[mySourceType]
# example: 2022-07-01T23:53:54 2022-07-01T23:53:54                                                                 TIME_FORMAT = %Y-%m-%dT%H:%M:%S
REPORT-default = sourcefields-default

transforms.conf
[sourcefields-default]
SOURCE_KEY = source                                                                           REGEX = /files/(.*?)/(.*?)/(.*?)/(.*?)\-(.*)
FORMAT = field1::$1 field2::$2 field3::$3 field4::$4 field5::$5

[gcusello]

Hi @zapping575,

could you share some sample of your logs that aren't separated?

Ciao.

Giuseppe

[zapping575]

Ciao @gcusello 

The timestamp of the events that are merged together (in this example) is

• 2023-01-31 10:40:01

This is how the event in question appears in the original file (some entries truncated for clarity):

Note that the first occurrence of the timestamp in question is on the second line.

2023-01-31 10:39:58 message1
2023-01-31 10:40:01 message2
2023-01-31 10:40:08 message3
2023-01-31 10:40:08 message4
2023-01-31 10:40:00 some message
2023-01-31 10:40:01 some message in between
2023-01-31 10:40:01 some message in between
2023-01-31 10:40:01 some message in between
2023-01-31 10:40:01 message5
2023-01-31 10:40:01 message5

This listing illustrates how the single, merged event appears in splunk search (with the timestamp mentioned above).

2023-01-31 10:40:01 message2
2023-01-31 10:40:01 message5
2023-01-31 10:40:01 message5
2023-01-31 10:40:05 some message from later

Sharing the original data is difficult so I am hoping that this condensed version will suffice. There are no anomalies regarding missing newlines on any of the offending events.

Regards,

[gcusello · Accepted solution]

Hi @zapping575,

try adding 

SHOULD_LINEMERGE = false

to you props.conf and this should be sufficient.

eventually add also to you props.conf:

TIME_PREFIX = ^

Ciao.

Giuseppe

[zapping575]

Hi @gcusello 

I cannot make that change on the productive system right away. But I have a dev environment where I just tested it.

The search for index=index_in_question linecount > 1 now returns zero results, so this solved the problem.

Thank you.

[gcusello]

Hi @zapping575,

good for you, see next time!

Ciao and happy splunking

Giuseppe

P.S.: Karma Points are appreciated 😉