Events are incomplete - sometimes

dimitriosDev — Tue, 20 Oct 2020 13:13:44 GMT

Hello Devs,

unfortunately my splunk events getting cut sometimes and only the end part (varies from last 1k to 10k lines) of the message appears in splunk.
This appears to happen only to large events, events with 5000+ lines or 100kb+ one liner events.
By sometimes i mean it occurs in about 30 out of 200k+ cases and if I reimport them by copying them again into a logfile they appear complete in splunk.

The Logfiles coming via a log4j2 Async FileAppender and uses sourcetype log4j-a

props.conf

[log4j-a]
TRUNCATE = 0
BREAK_ONLY_BEFORE = \d{4}.\d{2}.\d{2}\s+\d{2}:\d{2}:\d{2}\s+-\s+log=
MAX_EVENTS=50000

I tried to use time_before_close and multiline_event_extra_waittime in the inputs.conf but it got worse and other events which had not been incomplete before got corrupted.

Any Ideas what might be the problem?
Let me know if you need more information.

Best regards,

Dimitrios

topic Events are incomplete - sometimes in Getting Data In

Events are incomplete - sometimes