I tried this config and as I said before. It works however for sometimes there are some events which are truncated in the line date. If you execute this configuration via Data preview it works. But for any unknown reason the event is truncated some times (~20% of times) ... View more

Around 30~40 lines. Just included this option because if I search the errors in "_internal" I found some entries related to achieving the truncate limit. ... View more

Hello alemarzu. The version is 6.4 Below we have the log file to be read by splunk, the props and tranform files: LOG FILE: =============================================================== = JOB : R3BRP#DECOUPLE_NFE[(0006 01/06/18),(0AAAAAAAAAAIROIP)].CL_S09_IFIPD_DECOUPLE_NFE_R3BRP_01 = USER : XXX 631/S/*ATHOCO/IBM/AUTOMATION_COORD_HORTOLANDIA/ = JCLFILE : / -job IFIPD_DECOUPLE_NFE -user FF_PRO1 -i 23154800 -c a = Job Number: 47383116 = Sat 01/06/18 17:10:57 BRST +++ IBM Tivoli Workload Scheduler for Applications, method R3BATCH 8.5.0 (patchrev 1 - 16:42:24 Jun 13 2014) +++ is called with following parameters: +++ -t LJ -c R3BRP,SAPECCPINST1,ACSXTWS02 -n 172.22.8.248 -p 31111 -r 1971,1971 -s 0AAAAAAAAAAIROIP -d 20180106,1515196800 -l twsuser1 -o /amb/local/tws/sapeccpinst1/TWS/stdlist/2018.01.06/O47383116.1710 -j CL_S09_IFIPD_DECOUPLE_NFE_R3BRP_01,47383116 -- / -job IFIPD_DECOUPLE_NFE -user FF_PRO1 -i 23154800 -c a +++ EEWO1031I The Tivoli Workload Scheduler home directory was found: ./.. +++ EEWO1027I The RFC connection is established: (1) +++ EEWO1023I Started the R/3 job at the following date and time: 01/06-17:11 : IFIPD_DECOUPLE_NFE, 17105901 Sat Jan 6 17:10:57 2018 +++ EEWO1007I The job status has been set to EXEC: IFIPD_DECOUPLE_NFE 17105901 +++ EEWO1006I Job status: IFIPD_DECOUPLE_NFE 17105901 FINISHED +++ EEWO1061I Job IFIPD_DECOUPLE_NFE with job ID 17105901 was executed on SAP application server XXXXX_YYYYYY_PPPPPPP. +++ EEWO1048I Retrieving the joblog of a job:: IFIPD_DECOUPLE_NFE , 17105901 *** WARNING 914 *** EEWO0914W An internal error has occurred. Either the joblog or the job protocol for the following job does not exist: Job name: IFIPD_DECOUPLE_NFE Job ID: 17105901. *** WARNING 904 *** EEWO0904W The program could not copy the joblog to stdout. *** WARNING 914 *** EEWO0914W An internal error has occurred. Either the joblog or the job protocol for the following job does not exist: Job name: IFIPD_DECOUPLE_NFE Job ID: 17105901. +++ EEWO1012I BDC sessions are complete at: 01/06-17:11 : 0 +++ EEWO1017I The job completed normally at the following date and time: 01/06-17:11 = Exit Status : 0 = System Time (Seconds) : 0 Elapsed Time (Minutes) : 0 = User Time (Seconds) : 0 = Sat 01/06/18 17:11:32 BRST Props file: TRUNCATE = 1000000 BREAK_ONLY_BEFORE = (= User Time (Seconds) : \d+\n= \w{3} \d{2}\/\d{2}\/\d{2} \d{2}:\d{2}:\d{2} [A-Z]+) DATETIME_CONFIG = LINE_BREAKER = (= User Time (Seconds) : \d+\n= \w{3} \d{2}\/\d{2}\/\d{2} \d{2}:\d{2}:\d{2} [A-Z]+) NO_BINARY_CHECK = true SHOULD_LINEMERGE = false TIME_PREFIX = = [A-Z][a-z]+\s TIME_FORMAT = %m/%d/%y %H:%M:%S %Z SEDCMD-applychange01=s/[\r\n]\s*[A-z]+.+//g SEDCMD-applychange02=s/(**+.)//g SEDCMD-applychange04=s/(+++.)//g SEDCMD-applychange05=s/(==+[\r\n]*)//g TRANSFORMS-set= setNullJob,setParsingJob Transform file [setNullJob] REGEX = . DEST_KEY = queue FORMAT = nullQueue [setParsingJob] REGEX = R3BRP#DECOUPLE_NFE DEST_KEY = queue FORMAT = indexQueue From this log file, I need to extract 3 data: start/end time, and the app server. ... View more

Hello alemarzu, I just executed the below query and got 22 entries in the last 15 minutes (where I had 3 truncated events and 12 correct events) regarding to the component "DateParserVerbose" with the error "Failed to parse timestamp". Maybe it could explain the intermittence. However if I test the props config via Data Preview, it would not show the timestamp failure... index=_internal source=splunkd.log AggregatorMiningProcessor OR LineBreakingProcessor OR DateParserVerbose WARN ... View more

Hello alemarzu, Tried this configuration however the issue persists. Sometimes it is still truncating the indexed text. If you go via Data preview, it will show correctly the 9 lines. I dont understand why sometimes it is not following the correct way. just as curiosity: whenever the truncate happens, this occurs 2 ~ 3 times in sequence. I am using also SEDCMD and transform. Dont know if the priorities can impact in something. Thanks and regards, Danillo Pavan ... View more

Maybe it is a important information. When I try to use the Data preview tool, using the same log file and sourcetype, it is showing only the correct lines (without truncate the lines) however it is showing a WARN information: "Could not use strptime to parse timestamp from ": R3BRP#DECOUPLE_NFE[(0006 01/02/". "Failed to parse timestamp. Defaulting to file modtime." ... View more

Hello garethatiag, I have included this one also. It seems that it has decreased the number of times the event is being truncated, however is still happening. Sometimes (around 20% of the total of events) are still being truncated in the line date. ... View more

Some days ago, i was getting this error that I posted saying that it was not encountered the word that I have configured for the property "BREAK_ONLY_BEFORE". Now i am not facing any issue anymore, but the event is getting truncated sometimes incorrectly as I posted in the question. ... View more

I have included the property: "TRUNCATE = 0" in props file and still not work. Sometimes the file is truncated. ... View more

Hello Imaclean, I have executed the both queries ( for the component DataParserVerbose and LineBreakingProcessor ), but didnt find anything. For the search: index=_internal source=*splunkd.log component=LineBreakingProcessor and just found some ERROR entries related to the BREAK_ONLY_BEFORE property that I have configured to read entire file, but it happened just few days ago - now i dont have any entry for this search. "LineBreakingProcessor - Line breaking regex has no capturing groups: somethingjunk" Executing the below query, didnt return any entry index=_internal source=*splunkd.log component=DataParserVerbose The problem is that it is so intermitent, sometimes all entire the file is indexed correctly and sometimes it is truncated in a specific line containing date. I have already used the SEDCMD to replace the data format by a string, but even with this replacement the file is truncated sometimes. So strange... ... View more

Hello petercow, I have executed the below query: index=_internal source=*splunkd.log component=LineBreakingProcessor and just found some ERROR entries related to the BREAK_ONLY_BEFORE property that I have configured to read entire file, but it happened just few days ago - now i dont have any entry for this search. "LineBreakingProcessor - Line breaking regex has no capturing groups: somethingjunk" Executing the below query, didnt return any entry index=_internal source=*splunkd.log component=DataParserVerbose Please let me know if there is any other search command that I could run to try to find out the reasons... ... View more

Not sure if is the best option considering the performance, but I have included in the transform a REGEX expression that contains words which are keeping in the event after the SED entries. So just to resume: it is working in the below way: First the SED entries in props file replace strings with regular expression match (the unwanted information is replaced by null string) and then the entire event is filtered using the transform file via null quee. Many thanks for all support here!!! Danillo Pavan ... View more

I was taking a look at the below topic, I found that the solution was to include the REGEX expression in transform instead of props because it seems the "SED-" entries in props file are executed prior to TRANSFORMS- Is it correct? Could you please help me try to include these REGEX expressions in transform? https://answers.splunk.com/answers/108452/filter-events-and-use-sedcmd.html ... View more

Checking now, verified that I just had 4 new events indexed. All of them related to other JOB. It is strange. I believe that the event filtering is not working anymore. Props.file [logJobsSAP] TRANSFORMS-set= setNullJob,setParsingJob SHOULD_LINEMERGE=true BREAK_ONLY_BEFORE=SOMEJUNK SEDCMD-preserveJobexce=s/(++\s+)(.+ executed.+)/= \2/ SEDCMD-removepluslines=s/(\s*++.+)//g SEDCMD-removejunk=s/(==+[\r\n])//g SEDCMD-removejunk1=s/(=\s+\w+\s+:.+[\r\n])//g SEDCMD-removejunk2=s/[\r\n]\s[A-z]+.+//g TIME_PREFIX==\s+\w{3} TIME_FORMAT=%m/%d/%y %H:%M:%S %Z Transform file [setNullJob] REGEX = . DEST_KEY = queue FORMAT = nullQueue [setParsingJob] REGEX = 23154800 DEST_KEY = queue FORMAT = indexQueue ... View more

Just checked that 3 hours after the change, something happened :). It seems that now it is showing just the correct lines ! It excluded the rest of the lines, but the transform that I have created to filter the event is not working anymore. I have created it to show just the log files related to the JOB ID 23154800, before this change it was indexing just the log files related to this JOB. The others log files were being redirected to null queue. But now, after this changed, it seems this command is not reflecting. My props files contains: TRANSFORMS-set= setNullJob,setParsingJob and the in the transform files: [setNullJob] REGEX = . DEST_KEY = queue FORMAT = nullQueue [setParsingJob] REGEX = 23154800 DEST_KEY = queue FORMAT = indexQueue Is it possible to conciliate events filtering with lines filtering? Thanks and regards, Danillo Pavan ... View more

Yes, I agree. I have other case where the log file has more than 80 lines and I would like to keep just 5 or 6 lines, disregarding the others. Similar case to this one, but not able to proceed with this filtering... ... View more

I tried but nothing was indexed. No events was indexed after this changed. Important to mention that i am using also transform file to filter the events. Those logs are referred to some jobs, so i am using transform just to index the log files related to the job where the ID is 23154800. My current props file is: [XXXXXX] TRANSFORMS-set= setNullJob,setParsingJob SHOULD_LINEMERGE=true BREAK_ONLY_BEFORE=SOMEJUNK SEDCMD-preserveJobexce=s/(++\s+)(.+ executed.+)/= \2/ SEDCMD-removepluslines=s/(\s*++.+)//g SEDCMD-removejunk=s/(==+[\r\n])//g SEDCMD-removejunk1=s/(=\s+\w+\s+:.+[\r\n])//g SEDCMD-removejunk2=s/[\r\n]\s[A-z]+.+//g TIME_PREFIX==\s+\w{3} TIME_FORMAT=%m/%d/%y %H:%M:%S %Z And the transform file is: [setNullJob] REGEX = . DEST_KEY = queue FORMAT = nullQueue [setParsingJob] REGEX = 23154800 DEST_KEY = queue FORMAT = indexQueue ... View more

I really didnt note anything in particular that make these lines different from the others. Just want to select them because they have important information that i want to extract. All other rows are junk that I would like to delete. Initially i thought about using transform file to filter to null queue, however i believe that this mechanism is for filter entire events and not some lines within of the event. Using regex what is your idea? to use in props file or transform file? Thanks and regards, Danillo Pavan ... View more

Hello somesoni, sure. Below is one example of the content of the log file that is being read. For each event, I have below multilines: =============================================================== = JOB : R3BRP#DECOUPLE_NFE[(0006 12/29/17),(0AAAAAAAAAAIO5BA)].CL_S09_IFIPD_DECOUPLE_NFE_R3BRP_01 = USER : tws 631/S/COORD_HORTOLANDIA/ = JCLFILE : / -job IFIPD_DECOUPLE_NFE -user FF_PRO1 -i 23154800 -c a = Job Number: 9177522 = Fri 12/29/17 16:10:53 BRST +++ IBM Tivoli Workload Scheduler for Applications, method R3BATCH 8.5.0 (patchrev 1 - 16:42:24 Jun 13 2014) +++ is called with following parameters: +++ EEWO1031I The Tivoli Workload Scheduler home directory was found: ./.. +++ EEWO1027I The RFC connection is established: (1) +++ EEWO1023I Started the R/3 job at the following date and time: 12/29-16:10 : IFIPD_DECOUPLE_NFE, 16105502 Fri Dec 29 16:10:53 2017 +++ EEWO1007I The job status has been set to EXEC: IFIPD_DECOUPLE_NFE 16105502 +++ EEWO1006I Job status: IFIPD_DECOUPLE_NFE 16105502 FINISHED +++ EEWO1061I Job IFIPD_DECOUPLE_NFE with job ID 16105502 was executed on SAP application server XXXXXXXXXX. +++ EEWO1048I Retrieving the joblog of a job:: IFIPD_DECOUPLE_NFE , 16105502 *** WARNING 914 *** EEWO0914W An internal error has occurred. Either the joblog or the job protocol for the following job does not exist: Job name: IFIPD_DECOUPLE_NFE Job ID: 16105502. *** WARNING 904 *** EEWO0904W The program could not copy the joblog to stdout. *** WARNING 914 *** EEWO0914W An internal error has occurred. Either the joblog or the job protocol for the following job does not exist: Job name: IFIPD_DECOUPLE_NFE Job ID: 16105502. +++ EEWO1012I BDC sessions are complete at: 12/29-16:11 : 0 +++ EEWO1017I The job completed normally at the following date and time: 12/29-16:11 = Exit Status : 0 = System Time (Seconds) : 0 Elapsed Time (Minutes) : 0 = User Time (Seconds) : 0 = Fri 12/29/17 16:11:27 BRST And I just would like to have few lines in a single event. Just the below lines: = Job Number: 9177522 = Fri 12/29/17 16:10:53 BRST +++ EEWO1061I Job IFIPD_DECOUPLE_NFE with job ID 16105502 was executed on SAP application server XXXXXXXXXX = Exit Status : 0 = System Time (Seconds) : 0 Elapsed Time (Minutes) : 0 = User Time (Seconds) : 0 = Fri 12/29/17 16:11:27 BRST ... View more

I still didn´t find any solution for this case. Still need to trash certain lines and then create multiline events out of what is left... Any proposal?? ... View more

Any answer? ... View more

Any answer? ... View more