Line breaking not working properly - event is not visible until next will be sent

[NoSpaces]

Hello to everyone!
My question looks very dummy, but I really can't understand how I can resolve it.
So, what we are having step by step:
1. Some network device that sends an event via UDP directly to an indexer
2. Indexer receives message according to capture of wireshark
3. Then I'm trying to find this event on a searchhead, and I see nothing
4. Somehow I generate another event on the network device
5. Then I expect to see two events during the search, but I see only the previous one

This behavior is a little bit random but easy to reproduce with network devices that send events unfrequently.
And, additionally, I can easily detect wrong behavior because of the significant difference between _time and _indextime of those events.

A couple of words about indexer settings, props.conf on indexer looks like this, nothing special:

[cat_syslog]
DETERMINE_TIMESTAMP_DATE_WITH_SYSTEM_TIME = true
MAX_TIMESTAMP_LOOKAHEAD = 24
SHOULD_LINEMERGE = false
TIME_PREFIX = ^<\d{1,3}>\d+:\s+.*:\s+\d+:\s+

Overall, what I can assume.
1. According to my props.conf, indexer expecting to find default ([\r\n]+) to apply line-breaking rule and create single event
2. But for some reason fails in it
3. From this moment, the indexer waits until the next event
4. An, I don't know why,  but ([\r\n}+) appears in the next message

So, the question is, how to NOT wait until next event in this situation?
I also understand that I can't change the line-breaking rule because of very unrequent events.
And also, there are no special characters at the end of events because they look like this:

<172>153702: 172.21.0.13: 153696: Sep 13 16:30:50.797 RTZ: %RADIUS-4-RADIUS_ALIVE: RADIUS server 172.28.20.80:1812,1813 is being marked alive.
<174>153700: 172.21.0.13: 153694: Sep 13 16:30:30.714 RTZ: %RADIUS-6-SERVERALIVE: Group AAA_RADIUS: Radius server 172.21.20.80:1812,1813 is responding again (previously dead).
<173>153695: 172.21.0.13: 153689: Sep 13 16:25:05.626 RTZ: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet0/9, changed state to up

[PickleRick]

Adding to what @richgalloway already said - if you're receiving events on udp:// input each datagram is treated as separate event (haven't tried what happens if you have additional line breaker in the middle of your event).

What can be happening, especially since you're saying there's a big difference between _time and _indextime - there is either an unsynced clock somewhere or the timezone is wrongly parsed, configured or assumed. So your event does come in, does get parsed and indexed but gets indexed into the future and you don't see it in your web interface because for all the default time ranges there is an implicit "latest=now". And your events are later than that. So search for them with something like "earliest=-1h latest=+1d".

[NoSpaces]

I'm really not an expert in datagrams, but according my observation behavior, it is not true
If each datagram is separate event, it's not possible to see the same behavior because with "SHOULD_LINMERGE = false" events can be defined without LINE_BREAKER
>each datagram is treated as separate event

Returning to the _indextime versus _teme - it's just an addition in my case, because if your log rate is pretty low, you can see in realtime how events showing in search only after the next one

[PickleRick]

OK. That's interesting. I was pretty sure it worked as I wrote but it apparently doesn't

From my tests it seems that even if I specify a completely non-existent sourcetype, Splunk does break events on udp:// input on a syslog priority header. And indeed does ingest event only after receiving <PRI> from the next event. That's strange. And that's one more reason for _not_ listening for syslog directly on the indexer/HF.

EDIT:  I haven't tested it yet but I suppose it can have something to do with:

no_priority_stripping = <boolean>
* Whether or not the input strips <priority> syslog fields from events it
  receives over the syslog input.
* A value of "true" means the instance does NOT strip the <priority>
  syslog field from received events.
* NOTE: Do NOT set this setting if you want to strip <priority>.
* Default: false

no_appending_timestamp = <boolean>
* Whether or not to append a timestamp and host to received events.
* A value of "true" means the instance does NOT append a timestamp
  and host to received events.
* NOTE: Do NOT set this setting if you want to append timestamp and host
  to received events.
* Default: false

[NoSpaces]

I'm familiar with the settings above.
I use no_priority_stripping on all udp inputs because I have some logic with PRI codes processing.

[richgalloway]

The first error is in using an indexer as a UDP receiver.  That likely will result in data loss.  Recommended practice for the last several years is to send syslog events to a dedicated syslog receiver, such as syslog-ng or rsyslog.  Then use a Universal Forwarder to send the events from the syslog server to Splunk.

The current LINE_BREAKER setting says an event doesn't end until it finds a newline so the incoming text is held until that condition is met.  This can be overridden in file inputs, but not with UDP (another case for using a syslog server).  Try this line breaker and also add TIME_FORMAT.

LINE_BREAKER = ([\r\n]+).*$
TIME_FORMAT = %b %d %H:%M:%S.%3N

[NoSpaces]

One additional question.
Do you know any free products that can act like a syslog server (listen to udp\tcp and store files) but working on Windows?

[NoSpaces]

Thank you for your response.
I understand that using a dedicated syslog server is the best practice, but until this moment I haven't understood with which errors I can come across without it.

I tested your props.conf suggestion but still observe the same behavior that was described in the OP.