Hi
As I see many documents and comments here, Universal forwarder do not break line. with "LINE_BREAKER" in props.conf. It is the role of Indexer. This is what I am understanding.
But I tested it by myself, and I saw that Universal forwarder is able to do line breaking.
-props.conf
[test_srctype]
SHOULD_LINEMERGE = false
LINE_BREAKER = ([\r\n]+)
[monitor:///home/ec2-user/test.log]
disabled = false
index = test_idx
sourcetype = test_srctype
Is my understanding wrong?
covers this quite well along with How indexing works
Effectively the universal forwarder will only do work on the data in the structured parsing queue, this is in the cases that:
If it's not those circumstances then the LINE_BREAKER should not apply
Under "Using forwarding agents" there is no mention of indexing/parsing data, all event processing would normally occur on the first Splunk enterprise instance to receive the data
That said it is good practice to configure an EVENT_BREAKER on the universal forwarder if possible.
In your testing did you include a timestamp on each line?
covers this quite well along with How indexing works
Effectively the universal forwarder will only do work on the data in the structured parsing queue, this is in the cases that:
If it's not those circumstances then the LINE_BREAKER should not apply
Under "Using forwarding agents" there is no mention of indexing/parsing data, all event processing would normally occur on the first Splunk enterprise instance to receive the data
That said it is good practice to configure an EVENT_BREAKER on the universal forwarder if possible.
In your testing did you include a timestamp on each line?
@gjanders Thanks a lot. In terms of my testing, the LINE_BREAKER setting was applied by Indexer's default props.conf. So you are right. Line breaking has done by only indexer or heavy forwarder.
BTW, in the case of EVENT_BREAKER setting on universal forwarder, it is only related to LB. I mean.. with EVENT_BREAKER setting, line breaking is not possible on forwarder. It have LB to determine if where is the event boundary. Even though EVENT_BREAKER is enabled on universal forwarder, LINE_BREAKER on Indexer would be applied. Is my understanding correct?
I would greatly appreciated if you answer again. Thank you!
@brandy81that is correct, EVENT_BREAKER is related to line breaking.
There are details somewhere but without event breaker, if you have a rapidly updating file with a monitor:// stanza, Splunk cannot switch to a new backend server as it is unsure where the end of the event is.
Therefore it waits until the data stops for a few seconds and then switches (this can result in a file going to 1 indexer/backend server)
EVENT_BREAKER makes it clear when the event is over and when it can safely switch to another indexer/backend server...