Getting Data In
Highlighted

Inconsistent linebreaker behavior

Communicator

Hello all,

I have configured the props file to NOT break the event when encounters a new line with a date, however, sometimes the event is broken in the line containing the date and sometimes the event is not truncated. I don't understand the reason for different behaviors.

Props file:
SHOULDLINEMERGE=true
BREAK
ONLYBEFOREDATE=false
BREAKONLYBEFORE=SOMEJUNK
MAXTIMESTAMPLOOKAHEAD=450
TIMEPREFIX==\s+\w{3}
TIME
FORMAT=%m/%d/%y %H:%M:%S %Z

File that is being read:

= JOB : R3BRP#DECOUPLENFE[(0006 01/02/18),(0AAAAAAAAAAIO5BE)].CLS09IFIPDDECOUPLENFER3BRP01
= USER : tws 631/S/*ATHOCO/IBM/AUTOMATION
COORDHORTOLANDIA/
= JCLFILE : / -job IFIPD
DECOUPLENFE -user FFPRO1 -i 23154800 -c a
= Job Number: 43977410
= Tue 01/02/18 15:50:05 BRST
*** WARNING 914 *** EEWO0914W An internal error has occurred. Either the joblog or the job protocol for the following job does not exist:
*** WARNING 904 *** EEWO0904W The program could not copy the joblog to stdout.
*** WARNING 914 *** EEWO0914W An internal error has occurred. Either the joblog or the job protocol for the following job does not exist:
= Exit Status : 0
= System Time (Seconds) : 0 Elapsed Time (Minutes) : 0
= User Time (Seconds) : 0
= Tue 01/02/18 15:50:39 BRST

Sometimes I got the multiline event containing the 12 lines, but sometimes the event is truncated like below sample:

= JOB : R3BRP#DECOUPLENFE[(0006 01/02/18),(0AAAAAAAAAAIO5BE)].CLS09IFIPDDECOUPLENFER3BRP01
= USER : tws 631/S/*ATHOCO/IBM/AUTOMATION
COORDHORTOLANDIA/
= JCLFILE : / -job IFIPD
DECOUPLENFE -user FFPRO1 -i 23154800 -c a
= Job Number: 35391514
= Tue 01/02/18 15:51:10 BRST

I need to have all entire text log indexed (12 lines) and not only the 5 above lines. Dont know why for sometimes the event is broken in the line date.

Thanks and regard,
Danillo Pavan

0 Karma
Highlighted

Re: Inconsistent linebreaker behavior

Path Finder

Check the _internal index for sourectype "splunkd" where you're indexing. Look for 'ERROR' or 'WARN' for that sourcetype.

0 Karma
Highlighted

Re: Inconsistent linebreaker behavior

Communicator

Hello petercow, I have executed the below query:

index=_internal source=*splunkd.log component=LineBreakingProcessor

and just found some ERROR entries related to the BREAKONLYBEFORE property that I have configured to read entire file, but it happened just few days ago - now i dont have any entry for this search.

"LineBreakingProcessor - Line breaking regex has no capturing groups: somethingjunk"

Executing the below query, didnt return any entry

index=_internal source=*splunkd.log component=DataParserVerbose

Please let me know if there is any other search command that I could run to try to find out the reasons...

0 Karma
Highlighted

Re: Inconsistent linebreaker behavior

Path Finder

So are you saying you get the regex error when you get the bad line-breaking?

0 Karma
Highlighted

Re: Inconsistent linebreaker behavior

Communicator

Some days ago, i was getting this error that I posted saying that it was not encountered the word that I have configured for the property "BREAKONLYBEFORE". Now i am not facing any issue anymore, but the event is getting truncated sometimes incorrectly as I posted in the question.

0 Karma
Highlighted

Re: Inconsistent linebreaker behavior

SplunkTrust
SplunkTrust

You should use LINE_BREAKER rather than BREAK_ONLY_BEFORE. You should also set SHOULD_LINEMERGE = false

0 Karma
Highlighted

Re: Inconsistent linebreaker behavior

Communicator

I have included the property: "TRUNCATE = 0" in props file and still not work. Sometimes the file is truncated.

0 Karma
Highlighted

Re: Inconsistent linebreaker behavior

SplunkTrust
SplunkTrust

Don't do this..

0 Karma
Highlighted

Re: Inconsistent linebreaker behavior

Path Finder

Look within the _internal index for the answers and to get at the issue faster use:

These errors are the ones related to TIMEFORMAT or LINEBREAKER errors:

index=_internal source=*splunkd.log component=DataParserVerbose WARN OR ERROR

For some related to Line Breaking issues:

index=_internal source=*splunkd.log component=LineBreakingProcessor WARN OR ERROR

These are the ones that will be related to MAX_EVENTS (256 Lines by default) & TRUNCATE (10,000 bytes by default) which are some of the top two causes but there are many others...

A good 2016 Splunk .Conf preso (also one in '13 & '14) is the "Jiffy lube quick tune up for you Splunk environment":

https://conf.splunk.com/files/2016/slides/jiffy-lube-quick-tune-up-for-your-splunk-environment.pdf

0 Karma
Highlighted

Re: Inconsistent linebreaker behavior

Communicator

Hello Imaclean, I have executed the both queries ( for the component DataParserVerbose and LineBreakingProcessor ), but didnt find anything.

For the search: index=_internal source=*splunkd.log component=LineBreakingProcessor

and just found some ERROR entries related to the BREAKONLYBEFORE property that I have configured to read entire file, but it happened just few days ago - now i dont have any entry for this search.

"LineBreakingProcessor - Line breaking regex has no capturing groups: somethingjunk"

Executing the below query, didnt return any entry

index=_internal source=*splunkd.log component=DataParserVerbose

The problem is that it is so intermitent, sometimes all entire the file is indexed correctly and sometimes it is truncated in a specific line containing date. I have already used the SEDCMD to replace the data format by a string, but even with this replacement the file is truncated sometimes. So strange...

0 Karma