topic Re: Line break doesn't work in Getting Data In

Line break doesn't work

rjfv8205 — Wed, 30 Sep 2020 00:30:38 GMT

I have following configuration

props.conf

[Scheduler]
NO_BINARY_CHECK = true
SHOULD_LINEMERGE = true
category = Custom
pulldown_type = 1
disabled = false
BREAK_ONLY_BEFORE = INICIO REPORTE

But line breaking doesn't work correctly in event in bold.

5/13/19
12:44:41.000 PM
17109 <13/05/2019 - 12:44:41>==================== INICIO REPORTE ======================
17109 Query :(sta..SP_STA_MON_OBTIENE_TAREAS 17109,1)
17109 RESPUESTA DE TAREAS PENDIENTES
Ret :0, nFilas : 17, nCols :1
Error:()
17109 ID. Tarea :(669153) Periodo Tarea :(201905)
17109 Nombre Tarea :(Traduccion - Conversion Archivo) Path Tarea :(/redbanc/bin_STI/bin/TAREA_CONVIERTE_ARCHIVOS)
17109 NUm. Params :(11)
17109 i :14, tot :17,EJECUTAR :[/redbanc/bin_STI/bin/TAREA_CONVIERTE_ARCHIVOS]
17109 Arg 0:[/redbanc/bin_STI/bin/TAREA_CONVIERTE_ARCHIVOS]
17109 Arg 1:[669153]
17109 Arg 2:[201905]
17109 Arg 3:[/redbanc/sca/casillas_STI/data/0008/2019050001182737REN00203715220190511]
17109 Arg 4:[/redbanc/sca/casillas_STI/data/0008/2019050000669164REN00203715220190511]
17109 Arg 5:[/0008]
17109 Arg 6:[/redbanc/sca/casillas_STI/data/0002/2019050001182738REN00203715220190511.CTR]
17109 Arg 7:[1]
17109 Arg 8:[1]
17109 Arg 9:[1]
17109 Arg 10:[1]
17109 Arg 11:[0]
17109 Arg 12:[607:83:44:21 9102 31 yaM7372811509102]
17109 Arg 13:[70]
17109 Se libera memoria de parametros : 13/05/2019 - 12:44:41
17109 <13/05/2019 - 12:44:41>==================== FIN REPORTE =====================

5/13/19
12:44:41.000 PM
17109 <13/05/2019 - 12:44:41>==================== INICIO REPORTE ======================
17109 Query :(sta..SP_STA_MON_OBTIENE_TAREAS 17109,1)
17109 RESPUESTA DE TAREAS PENDIENTES
Ret :0, nFilas : 17, nCols :1
Error:()
17109 ID. Tarea :(669157) Periodo Tarea :(201905)
17109 Nombre Tarea :(Traduccion - Conversion Archivo) Path Tarea :(/redbanc/bin_STI/bin/TAREA_CONVIERTE_ARCHIVOS)
17109 NUm. Params :(11)
17109 i :14, tot :17,EJECUTAR :[/redbanc/bin_STI/bin/TAREA_CONVIERTE_ARCHIVOS]
17109 Arg 0:[/redbanc/bin_STI/bin/TAREA_CONVIERTE_ARCHIVOS]
17109 Arg 1:[669157]
17109 Arg 2:[201905]
17109 Arg 3:[/redbanc/sca/casillas_STI/data/0009/2019050001182739REN00203715220190512]
17109 Arg 4:[/redbanc/sca/casillas_STI/data/0009/2019050000669165REN00203715220190512]
17109 Arg 5:[/0009]
17109 Arg 6:[/redbanc/sca/casillas_STI/data/0000/2019050001182740REN00203715220190512.CTR]
17109 Arg 7:[1]
17109 Arg 8:[1]
17109 Arg 9:[1]
17109 Arg 10:[1]
17109 Arg 11:[0]
17109 Arg 12:[617:93:44:21 9102 31 yaM9372811509102]
17109 Arg 13:[71]
17109 Se libera memoria de parametros : 13/05/2019 - 12:44:41
17109 <13/05/2019 - 12:44:41>==================== FIN REPORTE =====================

5/13/19
12:44:41.000 PM
17109 <13/05/2019 - 12:44:41>==================== INICIO REPORTE ======================
17109 Query :(sta..SP_STA_MON_OBTIENE_TAREAS 17109,1)

5/13/19
12:44:41.000 PM
17109 RESPUESTA DE TAREAS PENDIENTES
Ret :0, nFilas : 1, nCols :1
Error:()
17109 NO HAY TAREA
17109 <13/05/2019 - 12:44:41>==================== FIN REPORTE =====================

5/13/19
12:44:44.000 PM
17109 <13/05/2019 - 12:44:44>==================== INICIO REPORTE ======================
17109 Query :(sta..SP_STA_MON_OBTIENE_TAREAS 17109,1)
17109 RESPUESTA DE TAREAS PENDIENTES
Ret :0, nFilas : 11, nCols :1
Error:()
17109 ID. Tarea :(669127) Periodo Tarea :(201905)
17109 Nombre Tarea :(Generacion de Archivo Aviso) Path Tarea :(/redbanc/bin_STI/bin/TAREA_GENERA_AVI)
17109 NUm. Params :(5)
17109 i :8, tot :11,EJECUTAR :[/redbanc/bin_STI/bin/TAREA_GENERA_AVI]
17109 Arg 0:[/redbanc/bin_STI/bin/TAREA_GENERA_AVI]
17109 Arg 1:[669127]
17109 Arg 2:[201905]
17109 Arg 3:[/redbanc/sca/casillas_STI/data/0005/2019050000669127REN00202715220190512.AVI]
17109 Arg 4:[REN00202715220190512]
17109 Arg 5:[cca777p]
17109 Arg 6:[/0005]
17109 Arg 7:[LA TRANSMISIÓN DEL ARCHIVOS REN00202715220190512 HA LLEGADO SATISFACTORIAMENTE.]
17109 Se libera memoria de parametros : 13/05/2019 - 12:44:44
17109 <13/05/2019 - 12:44:44>==================== FIN REPORTE =====================

Re: Line break doesn't work

somesoni2 — Tue, 14 May 2019 18:03:27 GMT

Give this a try

[Scheduler]
NO_BINARY_CHECK = true
SHOULD_LINEMERGE = false
LINE_BRAKER = ([\r\n]+)(?=\d+\s+\<[^\>]+\>\=+INICIO REPORTE)
TIME_PREFIX = ^\d+\s+\<
TIME_FORMAT = %d/%m/%Y - %H:%M:%S
MAX_TIMESTAMP_LOOKAHEAD = 21
category = Custom
pulldown_type = 1
disabled = false

Re: Line break doesn't work

rjfv8205 — Tue, 14 May 2019 18:26:02 GMT

Why should_linemerge=false?

I understand that should_linemerge=true for multi-line events?

Re: Line break doesn't work

rjfv8205 — Tue, 14 May 2019 20:15:02 GMT

@somesoni2 doesn't work. I tested and this happened

1/15/19
5:31:24.000 PM

3457 <15/01/2019 - 17:31:24>==================== INICIO REPORTE ======================
3457 ID Tarea :(844205)
3457 Nombre Archivo AVI :(/redbanc/sca/casillas_STI/data/0006/2019010000844205RPT028190115172140.AVI)
3457 Nombre Archivo Datos :(RPT028190115172140 )
3457 Casilla Origen :(cca777p )
Show all 144 lines

I have unique event with several lines

Re: Line break doesn't work

rmjharris — Wed, 30 Sep 2020 00:34:12 GMT

The space before "INICIO REPORTE" was missed and LINE_BREAKER is misspelled. Make this change to the config and it should work.

LINE_BRAKER = ([\r\n]+)(?=\d+\s+<[^>]+>=+INICIO REPORTE)
to
LINE_BREAKER = ([\r\n]+)(?=\d+\s+<[^>]+>=+\s+INICIO REPORTE)

If you are using LINE_BREAKER then you should set SHOULD_LINEMERGE = false. From the Splunk documentation on props.conf:

NOTE: You get a significant boost to processing speed when you use LINE_BREAKER to delimit multi-line events (as opposed to using SHOULD_LINEMERGE to reassemble individual lines into multi-line events).
- When using LINE_BREAKER to delimit events, SHOULD_LINEMERGE should be set to false, to ensure no further combination of delimited events occurs.
- Using LINE_BREAKER to delimit events is discussed in more detail in the documentation. Search the documentation for "configure event line breaking" for details.

Re: Line break doesn't work

rjfv8205 — Wed, 15 May 2019 13:12:23 GMT

@rmjharris Thanks work fine. Monitor if the line breaking work correctly

How do you explain how work this regex please?

In the other hand, why there are two capturing groups?

Sorry, i'm new in this.

Thank you in advance.

Re: Line break doesn't work

rmjharris — Wed, 30 Sep 2020 00:31:00 GMT

From the props.conf documentation:

LINE_BREAKER =
* Specifies a regex that determines how the raw text stream is broken into
initial events, before line merging takes place. (See the SHOULD_LINEMERGE
setting, below)
* Defaults to ([\r\n]+), meaning data is broken into an event for each line,
delimited by any number of carriage return or newline characters.
* The regex must contain a capturing group -- a pair of parentheses which
defines an identified subcomponent of the match.
* Wherever the regex matches, Splunk software considers the start of the first
capturing group to be the end of the previous event, and considers the end
of the first capturing group to be the start of the next event.
* The contents of the first capturing group are discarded, and will not be
present in any event. You are telling Splunk software that this text comes
between lines.

So the first capture group matches and discards the return/new line but then you need to identify the start of the event not just create a new event at each new line. Since Splunk discards the contents of the first capture group you need to create a second that won't be discarded.

In this case @somesoni2 wrote the second part as an assertion rather than a capture group. You can read about the difference here. You can remove the ?= and it should work as well.

If it's working for you make sure you mark this correct. @somesoni2 did 99% of the work I just corrected a minor mistake.