Deployment Architecture

Dose Universal forwarder do line breaking?

Explorer

Hi 

As I see many documents and comments here, Universal forwarder do not break line. with "LINE_BREAKER" in props.conf. It is the role of Indexer. This is what I am understanding.

But I tested it by myself, and I saw that Universal forwarder is able to do line breaking.

-props.conf

[test_srctype]

SHOULD_LINEMERGE = false

LINE_BREAKER = ([\r\n]+)

[monitor:///home/ec2-user/test.log]

disabled = false

index = test_idx

sourcetype = test_srctype

 

Is my understanding wrong?

Labels (1)
0 Karma
1 Solution

SplunkTrust
SplunkTrust

Where to put props (Aplura) 

covers this quite well along with How indexing works

Effectively the universal forwarder will only do work on the data in the structured parsing queue, this is in the cases that:

If it's not those circumstances then the LINE_BREAKER should not apply

Under "Using forwarding agents" there is no mention of indexing/parsing data, all event processing would normally occur on the first Splunk enterprise instance to receive the data

That said it is good practice to configure an EVENT_BREAKER on the universal forwarder if possible.

In your testing did you include a timestamp on each line?

View solution in original post

0 Karma

SplunkTrust
SplunkTrust

Where to put props (Aplura) 

covers this quite well along with How indexing works

Effectively the universal forwarder will only do work on the data in the structured parsing queue, this is in the cases that:

If it's not those circumstances then the LINE_BREAKER should not apply

Under "Using forwarding agents" there is no mention of indexing/parsing data, all event processing would normally occur on the first Splunk enterprise instance to receive the data

That said it is good practice to configure an EVENT_BREAKER on the universal forwarder if possible.

In your testing did you include a timestamp on each line?

View solution in original post

0 Karma

Explorer

@gjanders  Thanks a lot. In terms of my testing, the LINE_BREAKER setting was applied by Indexer's default props.conf. So you are right. Line breaking has done by only indexer or heavy forwarder.

BTW, in the case of EVENT_BREAKER setting on universal forwarder, it is only related to LB. I mean.. with EVENT_BREAKER setting, line breaking is not possible on forwarder. It have LB to determine if where is the event boundary. Even though EVENT_BREAKER is enabled on universal forwarder, LINE_BREAKER on Indexer would be applied. Is my understanding correct?

I would greatly appreciated if you answer again. Thank you!

0 Karma

SplunkTrust
SplunkTrust

@brandy81that is correct, EVENT_BREAKER is related to line breaking.

There are details somewhere but without event breaker, if you have a rapidly updating file with a monitor:// stanza, Splunk cannot switch to a new backend server as it is unsure where the end of the event is.

Therefore it waits until the data stops for a few seconds and then switches (this can result in a file going to 1 indexer/backend server)

EVENT_BREAKER makes it clear when the event is over and when it can safely switch to another indexer/backend server...

0 Karma