@PaveIP, interesting information.
Events I'm ingesting are indeed potentially multiline (95% of them are single line) and I break events based on the date. Here's the relevant props.conf stanza.
[db.command.trace]
SHOULD_LINEMERGE = false
LINE_BREAKER = ([\r\n]+)\w+\s+\w+\s+\d+\s+\d+:\d+:\d+\.\d+\s
#LINE_BREAKER_LOOKBEHIND = 4096
MAX_TIMESTAMP_LOOKAHEAD = 40
TIME_PREFIX = ^
#Wed Mar 25 04:05:00.978 2020
TIME_FORMAT = %a %b %e %H:%M:%S.%Q %Y
I get errors from indexers regarding date/time extraction although when I search for events, I don't see any problem. I have not configured event breaker on the UF but I believe I should, I'll give it a try.
But we are also ingesting many other source types, including multiline ones and we've never experienced such problem before so I was wondering how it works under normal situations.
Thanks for the test but I don't completely understand how it's done. Is your event multiline ? Yes, a slow inputs will generally not work well with multiline. Splunk will send an event as soon as it reaches "\n"+EOF , so you need to play with the following parameter in inputs.conf:
multiline_event_extra_waittime = <boolean>
* By default, the file monitor sends an event delimiter when:
* It reaches EOF of a file it monitors and
* The last character it reads is a newline.
* In some cases, it takes time for all lines of a multiple-line event to
arrive.
* Set to "true" to delay sending an event delimiter until the time that the
file monitor closes the file, as defined by the 'time_before_close' setting,
to allow all event lines to arrive.
* Default: false
... View more