ingest csv files containing multi line fields

[gcusello]

Hi at all,

I have to ingest a csv file where some fields are multivalue and multiline, something like this:

FIELD1;FIELD2;FIELD3;FIELD4;FIELD5
xxxx;yyyyy;"ppp
";"qqq
asd
asd
ert
www";qwerty

How can I do it? I tried in many ways but it fails everytime.

Ciao and Thanks.

Giuseppe

[richgalloway]

I've done that in the past and, IIRC, it just worked. Can you share your props?

[gcusello]

Hi @richgalloway ,

thanks for your help.

this is my last props.conf, I used also other tries (e.g. listing fields instead HEADER_FIELD_LINE_NUMBER) but with the same result:

[my_sourcetype]
BREAK_ONLY_BEFORE_DATE = 
DATETIME_CONFIG = CURRENT
FIELD_DELIMITER = ;
FIELD_QUOTE = "
INDEXED_EXTRACTIONS = csv
KV_MODE = none
LINE_BREAKER = ([\r\n]+)
NO_BINARY_CHECK = true
SHOULD_LINEMERGE = false
category = Structured
description = Processes
disabled = false
pulldown_type = 1
PREAMBLE_REGEX = ^XXX
HEADER_FIELD_LINE_NUMBER = 1

Ciao and thanks.

 Giuseppe

[richgalloway]

This is what I get when I onboard your data with those settings.

[gcusello]

Hi @richgalloway ,

when I try to ingest my logs I have an event for each row.

Maybe I could use SHOULD_LINEMERGE = true but I don't know how to divide events.

Ciao.

Giuseppe

[richgalloway]

I suspect we're each doing something differently to get the different results, but I'm not sure what that something is.

Using SHOULD_LINEMERGE = true might help, but would break fields that are intended to have multiple lines. Then, as you say, there's the matter of how to break events properly.