Splunk Enterprise

Universal Forwarder Multi-Line Event Line Breaking

raynold_peterso
Path Finder

Good morning all,

I have been beating my head against this issue for a week or more.  Let me give you the details.

We have one indexer and multiple Universal Forwarders in the field.  One of these forwarders I am running a scripted input to gather directory data for a file monitoring solution.

input.conf:

 

###### Scripted Input to monitor jpeg files
[script://.\bin\dircontents.bat]
disabled = 0
## Run once per minute
interval = 60
sourcetype = Script:dir_files
index = filewatch

 

dircontents.bat

 

@echo off
D:
cd /seed
dir /b

 

The forwarder gathers this data from the script:

 

24Aug2017.txt
24Jan2018.txt
28Jul2016.txt
28Jul2016.txt~
29Jan2018.txt
INCHARGE-AM-PM-AL.seedfile
INCHARGE-AM-PM-AZ.seedfile
INCHARGE-AM-PM-GA-FL.seedfile
INCHARGE-AM-PM.seedfile
MitchDRSite.list
rcp.list
TSM-seed.list

 

This data is one event with Multiple lines.  I want to bread on the line feeds.  That sounds simple enough.  

props.conf

 

[Script:dir_files]
SHOULD_LINEMERGE = false
LINE_BREAKER = ([\r\n]+)
MAX_EVENTS = 10000
TRUNCATE = 0

 

After I deploy the configs to the UF, the data starts coming in as a single event with multiple lines.  Very frustrating!!!

I have tried many things, changed my regex around and I just can not find the solution.  

Any help would be appreciated at this time.

Let me know what you think

Rcp

 

0 Karma
1 Solution

richgalloway
SplunkTrust
SplunkTrust
Have you tried putting that props.conf file on the indexer?
---
If this reply helps you, Karma would be appreciated.

View solution in original post

0 Karma

richgalloway
SplunkTrust
SplunkTrust
Have you tried putting that props.conf file on the indexer?
---
If this reply helps you, Karma would be appreciated.
0 Karma

raynold_peterso
Path Finder

Rich,

I did have that thought this morning but wanted to get my question in.  I will try that and see what happens.

I would rather split the events on the UF before indexing.  That way I do not have to restart the production Splunk instance.

I'll try the props.conf on the indexer and will report the outcome.

 

Rcp

0 Karma

raynold_peterso
Path Finder

Well, that worked as expected.  The data broke on the line feeds at the indexer level.

I would still like to know if the data can be split up at the UF before sending the data.

 

Rcp

0 Karma

richgalloway
SplunkTrust
SplunkTrust
UFs don't do line breaking.
---
If this reply helps you, Karma would be appreciated.
0 Karma

dexterpokta
Engager

I release this is old but Universal Forwarders do perform EVENT_BREAKER properties. 
It was brought in for
1. Better load balancing.
2. Line break tuning would be more efficient, E.G. multiple lines of the same event would not be sent to different indexers.

See the props.conf.spec in the Universal forwarder for "EVENT_BREAKER" for more details.

Get Updates on the Splunk Community!

Register to Attend BSides SPL 2022 - It's all Happening October 18!

Join like-minded individuals for technical sessions on everything Splunk!  This is a community-led and run ...

What's New in Splunk Cloud Platform 9.0.2208?!

Howdy!  We are happy to share the newest updates in Splunk Cloud Platform 9.0.2208! Analysts can benefit ...

Admin Console: A Single, Unified Interface for All Your Cloud Admin Needs

WATCH NOWJoin us to learn how the admin console can save you time and give you more control over the Splunk® ...