topic Re: Universal Forwarder Multi-Line Event Line Breaking in Splunk Enterprise

Universal Forwarder Multi-Line Event Line Breaking

raynold_peterso — Tue, 30 Jun 2020 13:14:54 GMT

Good morning all,

I have been beating my head against this issue for a week or more. Let me give you the details.

We have one indexer and multiple Universal Forwarders in the field. One of these forwarders I am running a scripted input to gather directory data for a file monitoring solution.

input.conf:

###### Scripted Input to monitor jpeg files [script://.\bin\dircontents.bat] disabled = 0 ## Run once per minute interval = 60 sourcetype = Script:dir_files index = filewatch

dircontents.bat

@echo off D: cd /seed dir /b

The forwarder gathers this data from the script:

24Aug2017.txt 24Jan2018.txt 28Jul2016.txt 28Jul2016.txt~ 29Jan2018.txt INCHARGE-AM-PM-AL.seedfile INCHARGE-AM-PM-AZ.seedfile INCHARGE-AM-PM-GA-FL.seedfile INCHARGE-AM-PM.seedfile MitchDRSite.list rcp.list TSM-seed.list

This data is one event with Multiple lines. I want to bread on the line feeds. That sounds simple enough.

props.conf

[Script:dir_files] SHOULD_LINEMERGE = false LINE_BREAKER = ([\r\n]+) MAX_EVENTS = 10000 TRUNCATE = 0

After I deploy the configs to the UF, the data starts coming in as a single event with multiple lines. Very frustrating!!!

I have tried many things, changed my regex around and I just can not find the solution.

Any help would be appreciated at this time.

Let me know what you think

Rcp

Re: Universal Forwarder Multi-Line Event Line Breaking

richgalloway — Tue, 30 Jun 2020 13:34:18 GMT

Have you tried putting that props.conf file on the indexer?

Re: Universal Forwarder Multi-Line Event Line Breaking

raynold_peterso — Tue, 30 Jun 2020 13:52:11 GMT

Rich,

I did have that thought this morning but wanted to get my question in. I will try that and see what happens.

I would rather split the events on the UF before indexing. That way I do not have to restart the production Splunk instance.

I'll try the props.conf on the indexer and will report the outcome.

Rcp

Re: Universal Forwarder Multi-Line Event Line Breaking

raynold_peterso — Tue, 30 Jun 2020 16:16:24 GMT

Well, that worked as expected. The data broke on the line feeds at the indexer level.

I would still like to know if the data can be split up at the UF before sending the data.

Rcp

Re: Universal Forwarder Multi-Line Event Line Breaking

richgalloway — Tue, 30 Jun 2020 16:36:52 GMT

UFs don't do line breaking.

Re: Universal Forwarder Multi-Line Event Line Breaking

dexterpokta — Tue, 16 Nov 2021 06:10:36 GMT

I release this is old but Universal Forwarders do perform EVENT_BREAKER properties.
It was brought in for
1. Better load balancing.
2. Line break tuning would be more efficient, E.G. multiple lines of the same event would not be sent to different indexers.

See the props.conf.spec in the Universal forwarder for "EVENT_BREAKER" for more details.