Universal forwarder doesn't forward

[rjfv8205]

Splunk forwarder doesn't forward logs correctly. Validate in original source and logs have movement.

Some events arrive incomplete.

Original:

    29920 Inicia Ejecucion
    29920 <05/06/2019 - 13:16:21>
==================== INICIO REPORTE ======================
    29920 ID. Tarea                  :(236155)
    29920 Periodo Tarea              :(201906)
    29920 Archivo Control            :(/redbanc/sca/casillas_STI/data/0001/2019060000425145IDEBBVA190605003.ACU.CTR)
    29920 Archivo Datos Input        :(/redbanc/sca/casillas_STI/data/0010/2019060000425144IDEBBVA190605003.ACU)
    29920 Archivo Datos Logico       :(IDEBBVA190605003.ACU)
    29920 Tamano Archivo             :(32760.0)
    29920 Delimitador Casilla Origen :(SI TIENE)
    29920 Linea Control :(IDEBBVA190605003.ACUBHIFATM             SERVIEXP            0000000728F0000000044AN                    )
    29920 Tipo Registro              :(Fijo)
    29920 Numero Registros           :(0000000728)
    29920 Largo Reg. o Peso Archivo  :(0000000044)
    29920 Archivo Datos        :(/redbanc/sca/casillas_STI/data/0010/2019060000425144IDEBBVA190605003.ACU)
    29920 Archivo Datos Logico :(IDEBBVA190605003.ACU)
    29920 Casilla Origen       :(BHIFATM)
    29920 Casilla Destino      :(SERVIEXP)
    29920 Numero Registros     :(0000000728)
    29920 Tipo Reg             :(F)
    29920 Numero de Bytes      :(0000000044)
    29920 Tipo Formato         :(ASCII)
    29920 Traduccion           :(Sin Traduccion)
    29920 Query : (sta..SP_STA_TAR_VALIDA_CONTROL_OUT 236155,'201906',0,'Revision conforme','IDEBBVA190605003.ACU','BHIFATM','SERVIEXP',728,'F',44,'A','N')
    29920 <05/06/2019 - 13:16:21>

====================  FIN REPORTE    =====================

Splunk:

6/5/19
1:17:06.000 PM  
573 Inicia Ejecucion
host =  puma source =   /redbanc/bin_STI/logs/valida_ctr.log sourcetype =   Valida_CTR
6/5/19
1:16:21.000 PM  
29920 Inicia Ejecucion
host =  puma source =   /redbanc/bin_STI/logs/valida_ctr.log sourcetype =   Valida_CTR
6/5/19
1:15:44.000 PM  
29241 Inicia Ejecucion
host =  puma source =   /redbanc/bin_STI/logs/valida_ctr.log sourcetype =   Valida_CTR
6/5/19
1:15:13.000 PM  
28542 Inicia Ejecucion
host =  puma source =   /redbanc/bin_STI/logs/valida_ctr.log sourcetype =   Valida_CTR

While others doesn't arrive.

Original:

    24930 <05/06/2019 - 13:19:57>
==================== INICIO REPORTE ======================

    24930 Query :(sta..SP_STA_MON_OBTIENE_TAREAS 24930,1)
    24930 RESPUESTA DE TAREAS PENDIENTES
    Ret :0, nFilas : 17, nCols :1
    Error:()
    24930 ID. Tarea     :(236302) Periodo Tarea :(201906)
    24930 Nombre Tarea  :(Traduccion - Conversion Archivo) Path Tarea    :(/redbanc/bin_STI/bin/TAREA_CONVIERTE_ARCHIVOS)
    24930 NUm. Params   :(11)
    24930 i :14, tot :17,EJECUTAR :[/redbanc/bin_STI/bin/TAREA_CONVIERTE_ARCHIVOS]
    24930 Arg 0:[/redbanc/bin_STI/bin/TAREA_CONVIERTE_ARCHIVOS]
    24930 Arg 1:[236302]
    24930 Arg 2:[201906]
    24930 Arg 3:[/redbanc/sca/casillas_STI/data/0009/2019060000425499ABOBBVA190605003.ACU]
    24930 Arg 4:[/redbanc/sca/casillas_STI/data/0010/2019060000236306ABOBBVA190605003.ACU]
    24930 Arg 5:[/0010]
    24930 Arg 6:[/redbanc/sca/casillas_STI/data/0001/2019060000425500ABOBBVA190605003.ACU.CTR]
    24930 Arg 7:[1]
    24930 Arg 8:[1]
    24930 Arg 9:[1]
    24930 Arg 10:[1]
    24930 Arg 11:[0]
    24930 Arg 12:[007:15:91:1  9102 5  nuJ994524609102]
    24930 Arg 13:[70]
    24930 Se libera memoria de parametros : 05/06/2019 - 13:19:57
    24930 <05/06/2019 - 13:19:57>
====================  FIN REPORTE    =====================

Splunk:

6/5/19
11:50:19.000 AM 
24930 RESPUESTA DE TAREAS PENDIENTES
Ret :0, nFilas : 11, nCols :1
Error:()
24930 ID. Tarea     :(228511) Periodo Tarea :(201906)
24930 Nombre Tarea  :(Generacion de Archivo Aviso) Path Tarea    :(/redbanc/bin_STI/bin/TAREA_GENERA_AVI)
Show all 17 lines
host =  puma source =   /redbanc/bin_STI/logs/agente_scheduler.log sourcetype = Scheduler
6/3/19
8:04:35.000 AM  
24930 RESPUESTA DE TAREAS PENDIENTES
Ret :0, nFilas : 17, nCols :1
Error:()
24930 ID. Tarea     :(43413) Periodo Tarea :(201906)
24930 Nombre Tarea  :(Traduccion - Conversion Archivo) Path Tarea    :(/redbanc/bin_STI/bin/TAREA_CONVIERTE_ARCHIVOS)
Show all 23 lines
host =  puma source =   /redbanc/bin_STI/logs/agente_scheduler.log sourcetype = Scheduler

Note the last event time (event incomplete too)

Restart forwarder and I don't get results

We have a cluster with 1 master, 2 peer nodes, 1 SH

Master props.conf

[Scheduler]
NO_BINARY_CHECK = true
SHOULD_LINEMERGE = true
category = Custom
pulldown_type = 1
disabled = false
BREAK_ONLY_BEFORE = INICIO REPORTE

[Genera_AVI]
NO_BINARY_CHECK = true
SHOULD_LINEMERGE = true
category = Custom
pulldown_type = 1
disabled = false
BREAK_ONLY_BEFORE = INICIO REPORTE

[Valida_Res]
NO_BINARY_CHECK = true
SHOULD_LINEMERGE = true
category = Custom
pulldown_type = 1
disabled = false
BREAK_ONLY_BEFORE = INICIO REPORTE

[Valida_CTR]
NO_BINARY_CHECK = true
SHOULD_LINEMERGE = true
category = Custom
pulldown_type = 1
disabled = false
BREAK_ONLY_BEFORE = INICIO REPORTE

UF inputs.conf

[monitor:///redbanc/bin_STI/logs/agente_scheduler.log]
disabled = false
index = sti
sourcetype = Scheduler
multiline_event_extra_waittime = true

[monitor:///redbanc/bin_STI/logs/genera_avi.log]
disabled = false
index = sti
sourcetype = Genera_AVI
multiline_event_extra_waittime = true

[monitor:///redbanc/bin_STI/logs/valida_ctr.log]
disabled = false
index = sti
sourcetype = Valida_CTR
multiline_event_extra_waittime = true

[woodcock]

You should use these settings in props.conf:

[YourSourcetypeHere]
SHOULD_LINEMERGE = false
LINE_BREAKER = (?:([\r\n]+)[^\r\n]+){2}[\r\n]+=+ INICIO REPORTE
TIME_PREFIX = \d+\s<
TIME_FORMAT = %d/%m/%Y - %H:%M:%S
MAX_TIMESTAMP_LOOKAHEAD = 22

If you are doing a sourcetype override/overwrite, you must use the ORIGINAL values NOT the new value, then you must deploy this to the first full instance(s) of Splunk that handles the events (usually either the HF-tier, if you use this, or your Indexer tier), restart all Splunk instances there, send in new events (old events will stay broken), then test using _index_earliest=-5m to be absolutely certain that you are only examining the newly indexed events.

[gcusello]

Hi rjfv8205,
at first, the options you listed must be inserted in props.conf and not in outputs.conf.

About the line breaker, it's a regex, so please, try to use the following option

LINE_BREAKER = \d+\sInicia Ejecucion

Then you have to identify timestamp in props.conf:

TIME_PREFIX = \d+\sInicia Ejecucion\s+\d+\s\<
TIME_FORMAT = %d/%m/%Y - %H:%M:%S

Remember that props.conf must be located on Indexer, you have clustered Indexers, this means that you have to deploy it by Master Node.

Bye.
Giuseppe

[rjfv8205]

Sorry error writing. Options list are in props.conf

Splunk days ago index fine with conf listed in my post because I asked

[gcusello]

Hi rjfv8205,
probably the problem is in timestamp: your Splunk correctly indexed until 31st of may and will restart to correctly index from 13rd of June because it hasn't any doubt about time format.
if you search your 1st of June logs in 6th of january logs, you can find them.
You have to correctly configure TIME_FORMAT parameter because Splunk uses American time (mm/dd/yyyy) and you have european time (dd/mm/yyyy).
Bye.
Giuseppe