Merge lines from rotated file

chclemence — Tue, 11 May 2021 15:21:39 GMT

Hello,

I have some difficulties to ingest properly logs from rotated file, where the rotation is fully handled by an application, without any settings about its log files.

In some cases, it rotates the log file in the middle of an event, and before the timestamp.

Like this:

file.log.1

[category][event][reason][host][timestamp]...

[category][event][reason]

Note: there is no new line or carriage return at tne end of the file

file.log

[host][timestamp]...

[category][event][reason][host][timestamp]...

I have some events with an additional line in plain text (no brackets), so at first I let the line merger do its job, but for the splitted line between the 2 log files, it does not work as the first line does not contain the timestamp, and it is added to the previous event.

So, I disabled line merging and started to build a line breaker, resulting in this:

(?:\[[^]]*\][\r\n]*){24}\[[^]]*\](?:[^\[\]]*)([\r\n]+)

This is because the log line is made of 25 blocks between brackets, and optionally a second line.

It works very good on a test file when the splitted line is in the same file, but once the 2 parts of the line are in different files ... it does not work anymore 😞

Do you have any advice on how I could handle that ?

topic Merge lines from rotated file in Getting Data In

Merge lines from rotated file