Getting Data In

Event breaking not working properly with the regex ([\r\n]+)

Path Finder

Event breaking not working properly with the below regex...

props.conf
LINE_BREAKER=([\r\n]+)

My Log data :

('Hi-Spec JDA','Stuck Pick Work', 'No Pick Work - R/C','0','Stuck Pick Work','2019-07-17 09:10:01');
('Hi-Spec JDA','Locking Sessions', 'None','0','Locking Sessions','2019-07-17 09:10:01');
('Hi-Spec JDA','Database Locks', 'None','0','Database Locks','2019-07-17 09:10:01');
('Hi-Spec JDA','Long Running Waves', 'None','0','Long Running Waves','2019-07-17 09:10:01');
('Hi-Spec JDA','Broken Pipe Error', 'No Errors','0','Broken Pipe Error','2019-07-17 09:10:01');
('Hi-Spec JDA','Invmov with Inspect Inventory', 'Attention: Movement with Inspect Inventory Found','1','Invmov with Inspect Inventory','2019-07-17 09:10:01');
('Hi-Spec JDA','Movement Pick to VLM', 'No Movement going to VLM from Pick','0','Movement Pick to VLM','2019-07-17 09:10:01');

Issue here is, couple of lines are coming under the same event.

From the above example... there are 7 new lines of logs and it should break into 7 events...
But it breaks into 6 events as shown below..

EVENT 1 : ('Hi-Spec JDA','Stuck Pick Work', 'No Pick Work - R/C','0','Stuck Pick Work','2019-07-17 09:10:01');

EVENT 2 : ('Hi-Spec JDA','Locking Sessions', 'None','0','Locking Sessions','2019-07-17 09:10:01');

EVENT 3 : ('Hi-Spec JDA','Database Locks', 'None','0','Database Locks','2019-07-17 09:10:01');

EVENT 4 : ('Hi-Spec JDA','Long Running Waves', 'None','0','Long Running Waves','2019-07-17 09:10:01');

EVENT 5 (The one having issue) : ('Hi-Spec JDA','Broken Pipe Error', 'No Errors','0','Broken Pipe Error','2019-07-17 09:10:01');
('Hi-Spec JDA','Invmov with Inspect Inventory', 'Attention: Movement with Inspect Inventory Found','1','Invmov with Inspect Inventory','2019-07-17 09:10:01');

EVENT 6 : ('Hi-Spec JDA','Movement Pick to VLM', 'No Movement going to VLM from Pick','0','Movement Pick to VLM','2019-07-17 09:10:01');

What I believe the issue in EVENT 6 has something to do with the length of the second line in EVENT 6 which is the longest.

Can somebody please help me out here on how to rectify the issue.....????

0 Karma
1 Solution

SplunkTrust
SplunkTrust

The existing line breaker should work. See if this works any better: LINE_BREAKER = ;([\r\n]+)\(.

---
If this reply helps you, an upvote would be appreciated.

View solution in original post

0 Karma

Path Finder

The real problem was that my event breaking which is contained in props.conf was configured in the splunk cloud.
Now, since we are talking about HF here, so the HF was parsing and event breaking the data by-passing the configuration that I did in splunk cloud which was causing the issue.

To fix the issue, I copied the props.conf file from the splunk cloud and put it inside the HF which resolved the issue.

NOTE : The inputs.conf file was already inside the HF.

0 Karma

Esteemed Legend

You should also set these:

SHOULD_LINEMERGE = false
EVENT_BREAKER = ([\r\n]+)
EVENT_BREAKER_ENABLE = true
0 Karma

Path Finder

just as FYI... the regex ([\r\n]+) works like a charm when I manually add the data file under the sourcetype... It fails when data starts coming in from the forwarder..

Also my inputs.conf file is inside the heavy forwarder and NOT in Splunk cloud and props.conf file is in splunk cloud .
Can this be an issue??

0 Karma

Esteemed Legend

Why did you accept the answer that you did? What was the real problem? I am not saying that mine was right/better but it seems like you discovered something completely different as the resolution to the problem and it helps us all learn when you share.

0 Karma

Path Finder

I accepted the answer because of the comment from @richgalloway
"Something must be different about the data when it comes in from the forwarder.

Inputs.conf should be on the HF and NOT in Splunk Cloud."

The real problem was that my event breaking which is contained in props.conf was configured in the splunk cloud.
Now, since we are talking about HF here, so the HF was parsing and event breaking the data by-passing the configuration that I did in splunk cloud which was causing the issue.

To fix the issue, I copied the props.conf file from the splunk cloud and put it inside the HF which resolved the issue.

NOTE : The inputs.conf file was already inside the HF.

Hope this resolves your query.

0 Karma

SplunkTrust
SplunkTrust

The existing line breaker should work. See if this works any better: LINE_BREAKER = ;([\r\n]+)\(.

---
If this reply helps you, an upvote would be appreciated.

View solution in original post

0 Karma

Path Finder

Thanks for the response richagalloway, but unfortunately the regex provided by you further complicates things.

0 Karma

Path Finder

just as FYI... the regex ([\r\n]+) works like a charm when I manualy add the data file under the sourcetype... It fails when data starts coming in from the forwarder..

Also my inputs.conf file is inside the heavy forwarder and NOT in Splunk cloud. can this be an issue??

0 Karma

SplunkTrust
SplunkTrust

Something must be different about the data when it comes in from the forwarder.

Inputs.conf should be on the HF and NOT in Splunk Cloud.

---
If this reply helps you, an upvote would be appreciated.
0 Karma
State of Splunk Careers

Access the Splunk Careers Report to see real data that shows how Splunk mastery increases your value and job satisfaction.

Find out what your skills are worth!