Deployment Architecture

Dose Universal forwarder do line breaking?

brandy81
Path Finder

Hi 

As I see many documents and comments here, Universal forwarder do not break line. with "LINE_BREAKER" in props.conf. It is the role of Indexer. This is what I am understanding.

But I tested it by myself, and I saw that Universal forwarder is able to do line breaking.

-props.conf

[test_srctype]

SHOULD_LINEMERGE = false

LINE_BREAKER = ([\r\n]+)

[monitor:///home/ec2-user/test.log]

disabled = false

index = test_idx

sourcetype = test_srctype

 

Is my understanding wrong?

Labels (1)
0 Karma
1 Solution

gjanders
SplunkTrust
SplunkTrust

Where to put props (Aplura) 

covers this quite well along with How indexing works

Effectively the universal forwarder will only do work on the data in the structured parsing queue, this is in the cases that:

If it's not those circumstances then the LINE_BREAKER should not apply

Under "Using forwarding agents" there is no mention of indexing/parsing data, all event processing would normally occur on the first Splunk enterprise instance to receive the data

That said it is good practice to configure an EVENT_BREAKER on the universal forwarder if possible.

In your testing did you include a timestamp on each line?

View solution in original post

0 Karma

gjanders
SplunkTrust
SplunkTrust

Where to put props (Aplura) 

covers this quite well along with How indexing works

Effectively the universal forwarder will only do work on the data in the structured parsing queue, this is in the cases that:

If it's not those circumstances then the LINE_BREAKER should not apply

Under "Using forwarding agents" there is no mention of indexing/parsing data, all event processing would normally occur on the first Splunk enterprise instance to receive the data

That said it is good practice to configure an EVENT_BREAKER on the universal forwarder if possible.

In your testing did you include a timestamp on each line?

0 Karma

brandy81
Path Finder

@gjanders  Thanks a lot. In terms of my testing, the LINE_BREAKER setting was applied by Indexer's default props.conf. So you are right. Line breaking has done by only indexer or heavy forwarder.

BTW, in the case of EVENT_BREAKER setting on universal forwarder, it is only related to LB. I mean.. with EVENT_BREAKER setting, line breaking is not possible on forwarder. It have LB to determine if where is the event boundary. Even though EVENT_BREAKER is enabled on universal forwarder, LINE_BREAKER on Indexer would be applied. Is my understanding correct?

I would greatly appreciated if you answer again. Thank you!

0 Karma

gjanders
SplunkTrust
SplunkTrust

@brandy81that is correct, EVENT_BREAKER is related to line breaking.

There are details somewhere but without event breaker, if you have a rapidly updating file with a monitor:// stanza, Splunk cannot switch to a new backend server as it is unsure where the end of the event is.

Therefore it waits until the data stops for a few seconds and then switches (this can result in a file going to 1 indexer/backend server)

EVENT_BREAKER makes it clear when the event is over and when it can safely switch to another indexer/backend server...

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...