I resolved this and have tested for over 75 polls now.
The trick is to use this mysterious on/off inputs.conf line called:
multiline_event_extra_waittime = true
There's really no documentation on it other than the comments in the inputs.conf factory file:
multiline_event_extra_waittime = [true|false]
*By default, Splunk Enterprise sends an event delimiter when (1) it reaches EOF of a file it monitors and (2) the last char it reads is a newline.
*In some cases, it takes time for all lines of a multiple-line event to arrive.
*Set to true to delay sending an event delimiter until the time that Splunk Enterprise closes the file, as defined by the time_before_close attribute, to allow all event lines to arrive.
*Default to false.
and so, my final solution for my data stream was:
props.conf
root@server1:/opt/splunk/etc/apps/search/local# more props.conf
[custom_source_type_poller_v1]
SHOULD_LINEMERGE = true
LINE_BREAKER = ([\r\n]+)(?=Start: \w+, \d+)
TIME_PREFIX = ^Start:\s
TIME_FORMAT = %a, %d %b %Y %H:%M:%S %z
MAX_TIMESTAMP_LOOKAHEAD = 31
NO_BINARY_CHECK = true
category = Custom
pulldown_type = 1
disabled = false
inputs.conf
root@server1:/opt/splunk/etc/apps/search/local# more inputs.conf
[monitor:///data/poller/poller_v1.log]
disabled = false
index = poller_index
sourcetype = custom_source_type_poller_v1
time_before_close = 15
multiline_event_extra_waittime = true
... View more