Getting Data In

Event breaking not working properly with the regex ([\r\n]+)

soumdey
Path Finder

Event breaking not working properly with the below regex...

props.conf
LINE_BREAKER=([\r\n]+)

My Log data :

('Hi-Spec JDA','Stuck Pick Work', 'No Pick Work - R/C','0','Stuck Pick Work','2019-07-17 09:10:01');
('Hi-Spec JDA','Locking Sessions', 'None','0','Locking Sessions','2019-07-17 09:10:01');
('Hi-Spec JDA','Database Locks', 'None','0','Database Locks','2019-07-17 09:10:01');
('Hi-Spec JDA','Long Running Waves', 'None','0','Long Running Waves','2019-07-17 09:10:01');
('Hi-Spec JDA','Broken Pipe Error', 'No Errors','0','Broken Pipe Error','2019-07-17 09:10:01');
('Hi-Spec JDA','Invmov with Inspect Inventory', 'Attention: Movement with Inspect Inventory Found','1','Invmov with Inspect Inventory','2019-07-17 09:10:01');
('Hi-Spec JDA','Movement Pick to VLM', 'No Movement going to VLM from Pick','0','Movement Pick to VLM','2019-07-17 09:10:01');

Issue here is, couple of lines are coming under the same event.

From the above example... there are 7 new lines of logs and it should break into 7 events...
But it breaks into 6 events as shown below..

EVENT 1 : ('Hi-Spec JDA','Stuck Pick Work', 'No Pick Work - R/C','0','Stuck Pick Work','2019-07-17 09:10:01');

EVENT 2 : ('Hi-Spec JDA','Locking Sessions', 'None','0','Locking Sessions','2019-07-17 09:10:01');

EVENT 3 : ('Hi-Spec JDA','Database Locks', 'None','0','Database Locks','2019-07-17 09:10:01');

EVENT 4 : ('Hi-Spec JDA','Long Running Waves', 'None','0','Long Running Waves','2019-07-17 09:10:01');

EVENT 5 (The one having issue) : ('Hi-Spec JDA','Broken Pipe Error', 'No Errors','0','Broken Pipe Error','2019-07-17 09:10:01');
('Hi-Spec JDA','Invmov with Inspect Inventory', 'Attention: Movement with Inspect Inventory Found','1','Invmov with Inspect Inventory','2019-07-17 09:10:01');

EVENT 6 : ('Hi-Spec JDA','Movement Pick to VLM', 'No Movement going to VLM from Pick','0','Movement Pick to VLM','2019-07-17 09:10:01');

What I believe the issue in EVENT 6 has something to do with the length of the second line in EVENT 6 which is the longest.

Can somebody please help me out here on how to rectify the issue.....????

0 Karma
1 Solution

richgalloway
SplunkTrust
SplunkTrust

The existing line breaker should work. See if this works any better: LINE_BREAKER = ;([\r\n]+)\(.

---
If this reply helps you, Karma would be appreciated.

View solution in original post

0 Karma

soumdey
Path Finder

The real problem was that my event breaking which is contained in props.conf was configured in the splunk cloud.
Now, since we are talking about HF here, so the HF was parsing and event breaking the data by-passing the configuration that I did in splunk cloud which was causing the issue.

To fix the issue, I copied the props.conf file from the splunk cloud and put it inside the HF which resolved the issue.

NOTE : The inputs.conf file was already inside the HF.

0 Karma

woodcock
Esteemed Legend

You should also set these:

SHOULD_LINEMERGE = false
EVENT_BREAKER = ([\r\n]+)
EVENT_BREAKER_ENABLE = true
0 Karma

soumdey
Path Finder

just as FYI... the regex ([\r\n]+) works like a charm when I manually add the data file under the sourcetype... It fails when data starts coming in from the forwarder..

Also my inputs.conf file is inside the heavy forwarder and NOT in Splunk cloud and props.conf file is in splunk cloud .
Can this be an issue??

0 Karma

woodcock
Esteemed Legend

Why did you accept the answer that you did? What was the real problem? I am not saying that mine was right/better but it seems like you discovered something completely different as the resolution to the problem and it helps us all learn when you share.

0 Karma

soumdey
Path Finder

I accepted the answer because of the comment from @richgalloway
"Something must be different about the data when it comes in from the forwarder.

Inputs.conf should be on the HF and NOT in Splunk Cloud."

The real problem was that my event breaking which is contained in props.conf was configured in the splunk cloud.
Now, since we are talking about HF here, so the HF was parsing and event breaking the data by-passing the configuration that I did in splunk cloud which was causing the issue.

To fix the issue, I copied the props.conf file from the splunk cloud and put it inside the HF which resolved the issue.

NOTE : The inputs.conf file was already inside the HF.

Hope this resolves your query.

0 Karma

richgalloway
SplunkTrust
SplunkTrust

The existing line breaker should work. See if this works any better: LINE_BREAKER = ;([\r\n]+)\(.

---
If this reply helps you, Karma would be appreciated.
0 Karma

soumdey
Path Finder

Thanks for the response richagalloway, but unfortunately the regex provided by you further complicates things.

0 Karma

soumdey
Path Finder

just as FYI... the regex ([\r\n]+) works like a charm when I manualy add the data file under the sourcetype... It fails when data starts coming in from the forwarder..

Also my inputs.conf file is inside the heavy forwarder and NOT in Splunk cloud. can this be an issue??

0 Karma

richgalloway
SplunkTrust
SplunkTrust

Something must be different about the data when it comes in from the forwarder.

Inputs.conf should be on the HF and NOT in Splunk Cloud.

---
If this reply helps you, Karma would be appreciated.
0 Karma
Get Updates on the Splunk Community!

New Dates, New City: Save the Date for .conf25!

Wake up, babe! New .conf25 dates AND location just dropped!! That's right, this year, .conf25 is taking place ...

Introduction to Splunk Observability Cloud - Building a Resilient Hybrid Cloud

Introduction to Splunk Observability Cloud - Building a Resilient Hybrid Cloud  In today’s fast-paced digital ...

Observability protocols to know about

Observability protocols define the specifications or formats for collecting, encoding, transporting, and ...