topic Re: Inconsistent linebreaker behavior in Getting Data In

Inconsistent linebreaker behavior

danillopavan — Tue, 29 Sep 2020 17:28:05 GMT

Hello all,

I have configured the props file to NOT break the event when encounters a new line with a date, however, sometimes the event is broken in the line containing the date and sometimes the event is not truncated. I don't understand the reason for different behaviors.

Props file:
SHOULD_LINEMERGE=true
BREAK_ONLY_BEFORE_DATE=false
BREAK_ONLY_BEFORE=SOMEJUNK
MAX_TIMESTAMP_LOOKAHEAD=450
TIME_PREFIX==\s+\w{3}
TIME_FORMAT=%m/%d/%y %H:%M:%S %Z

File that is being read:

= JOB : R3BRP#DECOUPLE_NFE[(0006 01/02/18),(0AAAAAAAAAAIO5BE)].CL_S09_IFIPD_DECOUPLE_NFE_R3BRP_01
= USER : tws 631/S/ATHOCO/IBM/AUTOMATION_COORD_HORTOLANDIA/
= JCLFILE : / -job IFIPD_DECOUPLE_NFE -user FF_PRO1 -i 23154800 -c a
= Job Number: 43977410
= Tue 01/02/18 15:50:05 BRST
** WARNING 914 *** EEWO0914W An internal error has occurred. Either the joblog or the job protocol for the following job does not exist:
*** WARNING 904 *** EEWO0904W The program could not copy the joblog to stdout.
*** WARNING 914 *** EEWO0914W An internal error has occurred. Either the joblog or the job protocol for the following job does not exist:
= Exit Status : 0
= System Time (Seconds) : 0 Elapsed Time (Minutes) : 0
= User Time (Seconds) : 0
= Tue 01/02/18 15:50:39 BRST

Sometimes I got the multiline event containing the 12 lines, but sometimes the event is truncated like below sample:

= JOB : R3BRP#DECOUPLE_NFE[(0006 01/02/18),(0AAAAAAAAAAIO5BE)].CL_S09_IFIPD_DECOUPLE_NFE_R3BRP_01
= USER : tws 631/S/*ATHOCO/IBM/AUTOMATION_COORD_HORTOLANDIA/
= JCLFILE : / -job IFIPD_DECOUPLE_NFE -user FF_PRO1 -i 23154800 -c a
= Job Number: 35391514
= Tue 01/02/18 15:51:10 BRST

I need to have all entire text log indexed (12 lines) and not only the 5 above lines. Dont know why for sometimes the event is broken in the line date.

Thanks and regard,
Danillo Pavan

Re: Inconsistent linebreaker behavior

petercow — Thu, 04 Jan 2018 02:08:58 GMT

Check the _internal index for sourectype "splunkd" where you're indexing. Look for 'ERROR' or 'WARN' for that sourcetype.

Re: Inconsistent linebreaker behavior

lmaclean — Tue, 29 Sep 2020 17:29:10 GMT

Look within the _internal index for the answers and to get at the issue faster use:

These errors are the ones related to TIME_FORMAT or LINE_BREAKER errors:

index=_internal source=*splunkd.log component=DataParserVerbose WARN OR ERROR

For some related to Line Breaking issues:

index=_internal source=*splunkd.log component=LineBreakingProcessor WARN OR ERROR

These are the ones that will be related to MAX_EVENTS (256 Lines by default) & TRUNCATE (10,000 bytes by default) which are some of the top two causes but there are many others...

A good 2016 Splunk .Conf preso (also one in '13 & '14) is the "Jiffy lube quick tune up for you Splunk environment":

https://conf.splunk.com/files/2016/slides/jiffy-lube-quick-tune-up-for-your-splunk-environment.pdf

Re: Inconsistent linebreaker behavior

danillopavan — Tue, 29 Sep 2020 17:29:20 GMT

Hello petercow, I have executed the below query:

index=_internal source=*splunkd.log component=LineBreakingProcessor

and just found some ERROR entries related to the BREAK_ONLY_BEFORE property that I have configured to read entire file, but it happened just few days ago - now i dont have any entry for this search.

"LineBreakingProcessor - Line breaking regex has no capturing groups: somethingjunk"

Executing the below query, didnt return any entry

index=_internal source=*splunkd.log component=DataParserVerbose

Please let me know if there is any other search command that I could run to try to find out the reasons...

Re: Inconsistent linebreaker behavior

danillopavan — Tue, 29 Sep 2020 17:29:23 GMT

Hello Imaclean, I have executed the both queries ( for the component DataParserVerbose and LineBreakingProcessor ), but didnt find anything.

For the search: index=_internal source=*splunkd.log component=LineBreakingProcessor

and just found some ERROR entries related to the BREAK_ONLY_BEFORE property that I have configured to read entire file, but it happened just few days ago - now i dont have any entry for this search.

"LineBreakingProcessor - Line breaking regex has no capturing groups: somethingjunk"

Executing the below query, didnt return any entry

index=_internal source=*splunkd.log component=DataParserVerbose

The problem is that it is so intermitent, sometimes all entire the file is indexed correctly and sometimes it is truncated in a specific line containing date. I have already used the SEDCMD to replace the data format by a string, but even with this replacement the file is truncated sometimes. So strange...

Re: Inconsistent linebreaker behavior

danillopavan — Fri, 05 Jan 2018 15:25:32 GMT

I have included the property: "TRUNCATE = 0" in props file and still not work. Sometimes the file is truncated.

Re: Inconsistent linebreaker behavior

petercow — Fri, 05 Jan 2018 15:27:52 GMT

So are you saying you get the regex error when you get the bad line-breaking?

Re: Inconsistent linebreaker behavior

danillopavan — Tue, 29 Sep 2020 17:29:59 GMT

Some days ago, i was getting this error that I posted saying that it was not encountered the word that I have configured for the property "BREAK_ONLY_BEFORE". Now i am not facing any issue anymore, but the event is getting truncated sometimes incorrectly as I posted in the question.

Re: Inconsistent linebreaker behavior

gjanders — Sat, 06 Jan 2018 07:10:14 GMT

Perhaps you could try:

SHOULD_LINEMERGE = false
MAX_TIMESTAMP_LOOKAHEAD = 40
TIME_PREFIX=\=\s+\w{3}
TIME_FORMAT=%m/%d/%y %H:%M:%S %Z
LINE_BREAKER = (\n)NOBREAKINGPLEASE

Assuming this file is always 12 lines because the above will never create a break.

Re: Inconsistent linebreaker behavior

danillopavan — Sat, 06 Jan 2018 16:16:01 GMT

Hello garethatiag,

I have included this one also. It seems that it has decreased the number of times the event is being truncated, however is still happening. Sometimes (around 20% of the total of events) are still being truncated in the line date.

Re: Inconsistent linebreaker behavior

danillopavan — Sat, 06 Jan 2018 16:29:11 GMT

Maybe it is a important information. When I try to use the Data preview tool, using the same log file and sourcetype, it is showing only the correct lines (without truncate the lines) however it is showing a WARN information:
"Could not use strptime to parse timestamp from ": R3BRP#DECOUPLE_NFE[(0006 01/02/".
"Failed to parse timestamp. Defaulting to file modtime."

Re: Inconsistent linebreaker behavior

gjanders — Tue, 29 Sep 2020 17:34:41 GMT

If your using the LINE_BREAKER than the TRUNCATE setting should apply based on the amount of data, so you could increase that to avoid truncation, the splunkd log file should have a WARN or ERROR around the time of the issue if this is the case.

If your using the BREAK_ONLY_BEFORE_DATE (the default) then the parsing of the date matters.

Re: Inconsistent linebreaker behavior

alemarzu — Mon, 08 Jan 2018 15:19:04 GMT

Hello there @danillopavan

Try something like this.
props.conf

 [your_sourcetype]
 BREAK_ONLY_BEFORE = (= User Time \(Seconds\) : \d+\n= \w{3} \d{2}\/\d{2}\/\d{2} \d{2}:\d{2}:\d{2} [A-Z]+)
 DATETIME_CONFIG = 
 LINE_BREAKER = (= User Time \(Seconds\) : \d+\n= \w{3} \d{2}\/\d{2}\/\d{2} \d{2}:\d{2}:\d{2} [A-Z]+)
 NO_BINARY_CHECK = true
 SHOULD_LINEMERGE = false
 TIME_PREFIX = = [A-Z][a-z]+\s

Hope it helps.

Re: Inconsistent linebreaker behavior

danillopavan — Mon, 08 Jan 2018 16:46:03 GMT

Hello alemarzu,

Tried this configuration however the issue persists. Sometimes it is still truncating the indexed text. If you go via Data preview, it will show correctly the 9 lines. I dont understand why sometimes it is not following the correct way. just as curiosity: whenever the truncate happens, this occurs 2 ~ 3 times in sequence.

I am using also SEDCMD and transform. Dont know if the priorities can impact in something.

Thanks and regards,
Danillo Pavan

Re: Inconsistent linebreaker behavior

danillopavan — Mon, 08 Jan 2018 20:36:11 GMT

Hello alemarzu,

I just executed the below query and got 22 entries in the last 15 minutes (where I had 3 truncated events and 12 correct events) regarding to the component "DateParserVerbose" with the error "Failed to parse timestamp". Maybe it could explain the intermittence. However if I test the props config via Data Preview, it would not show the timestamp failure...

index=_internal source=splunkd.log AggregatorMiningProcessor OR LineBreakingProcessor OR DateParserVerbose WARN

Re: Inconsistent linebreaker behavior

alemarzu — Tue, 09 Jan 2018 15:50:37 GMT

Hi @danillopavan

Thats weird, I'm running Splunk 7.x.1 and its working, what version are you?

Try adding the TIME_FORMAT to your props.conf perhaps this might help and please could you share your actual props & transforms files ?

KR.

Re: Inconsistent linebreaker behavior

danillopavan — Tue, 29 Sep 2020 17:35:59 GMT

Hello alemarzu. The version is 6.4

Below we have the log file to be read by splunk, the props and tranform files:

LOG FILE:

===============================================================
= JOB : R3BRP#DECOUPLE_NFE[(0006 01/06/18),(0AAAAAAAAAAIROIP)].CL_S09_IFIPD_DECOUPLE_NFE_R3BRP_01
= USER : XXX 631/S/*ATHOCO/IBM/AUTOMATION_COORD_HORTOLANDIA/
= JCLFILE : / -job IFIPD_DECOUPLE_NFE -user FF_PRO1 -i 23154800 -c a
= Job Number: 47383116

= Sat 01/06/18 17:10:57 BRST

+++ IBM Tivoli Workload Scheduler for Applications, method R3BATCH 8.5.0 (patchrev 1 - 16:42:24 Jun 13 2014)
+++ is called with following parameters:
+++ -t LJ -c R3BRP,SAPECCPINST1,ACSXTWS02 -n 172.22.8.248 -p 31111 -r 1971,1971 -s 0AAAAAAAAAAIROIP -d 20180106,1515196800 -l twsuser1 -o /amb/local/tws/sapeccpinst1/TWS/stdlist/2018.01.06/O47383116.1710 -j CL_S09_IFIPD_DECOUPLE_NFE_R3BRP_01,47383116 -- / -job IFIPD_DECOUPLE_NFE -user FF_PRO1 -i 23154800 -c a
+++ EEWO1031I The Tivoli Workload Scheduler home directory was found: ./..
+++ EEWO1027I The RFC connection is established: (1)
+++ EEWO1023I Started the R/3 job at the following date and time: 01/06-17:11 : IFIPD_DECOUPLE_NFE, 17105901

Sat Jan 6 17:10:57 2018
+++ EEWO1007I The job status has been set to EXEC: IFIPD_DECOUPLE_NFE 17105901
+++ EEWO1006I Job status: IFIPD_DECOUPLE_NFE 17105901 FINISHED
+++ EEWO1061I Job IFIPD_DECOUPLE_NFE with job ID 17105901 was executed on SAP application server XXXXX_YYYYYY_PPPPPPP.

+++ EEWO1048I Retrieving the joblog of a job:: IFIPD_DECOUPLE_NFE , 17105901

*** WARNING 914 *** EEWO0914W An internal error has occurred. Either the joblog or the job protocol for the following job does not exist:
Job name: IFIPD_DECOUPLE_NFE
Job ID: 17105901.

*** WARNING 904 *** EEWO0904W The program could not copy the joblog to stdout.

*** WARNING 914 *** EEWO0914W An internal error has occurred. Either the joblog or the job protocol for the following job does not exist:
Job name: IFIPD_DECOUPLE_NFE
Job ID: 17105901.
+++ EEWO1012I BDC sessions are complete at: 01/06-17:11 : 0

+++ EEWO1017I The job completed normally at the following date and time: 01/06-17:11

= Exit Status : 0
= System Time (Seconds) : 0 Elapsed Time (Minutes) : 0
= User Time (Seconds) : 0

= Sat 01/06/18 17:11:32 BRST

Props file:
TRUNCATE = 1000000
BREAK_ONLY_BEFORE = (= User Time (Seconds) : \d+\n= \w{3} \d{2}\/\d{2}\/\d{2} \d{2}:\d{2}:\d{2} [A-Z]+)
DATETIME_CONFIG =
LINE_BREAKER = (= User Time (Seconds) : \d+\n= \w{3} \d{2}\/\d{2}\/\d{2} \d{2}:\d{2}:\d{2} [A-Z]+)
NO_BINARY_CHECK = true
SHOULD_LINEMERGE = false
TIME_PREFIX = = [A-Z][a-z]+\s
TIME_FORMAT = %m/%d/%y %H:%M:%S %Z
SEDCMD-applychange01=s/[\r\n]\s*[A-z]+.+//g
SEDCMD-applychange02=s/(**+.)//g
SEDCMD-applychange04=s/(+++.)//g
SEDCMD-applychange05=s/(==+[\r\n]*)//g
TRANSFORMS-set= setNullJob,setParsingJob

Transform file
[setNullJob]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[setParsingJob]
REGEX = R3BRP#DECOUPLE_NFE
DEST_KEY = queue
FORMAT = indexQueue

From this log file, I need to extract 3 data: start/end time, and the app server.

Re: Inconsistent linebreaker behavior

skoelpin — Tue, 09 Jan 2018 20:51:20 GMT

@garethatiag is 100% correct. Once these base configs are applied then it will work correctly.

I try to stay away from the UI onboarding option and just edit props.conf directly.

Re: Inconsistent linebreaker behavior

skoelpin — Tue, 09 Jan 2018 20:52:34 GMT

You should use LINE_BREAKER rather than BREAK_ONLY_BEFORE. You should also set SHOULD_LINEMERGE = false

Re: Inconsistent linebreaker behavior

skoelpin — Tue, 09 Jan 2018 20:52:53 GMT

Don't do this..