topic Re: ingest csv files containing multi line fields in Getting Data In

ingest csv files containing multi line fields

gcusello — Thu, 09 Jul 2020 15:31:04 GMT

Hi at all,

I have to ingest a csv file where some fields are multivalue and multiline, something like this:

FIELD1;FIELD2;FIELD3;FIELD4;FIELD5 xxxx;yyyyy;"ppp ";"qqq asd asd ert www";qwerty

How can I do it? I tried in many ways but it fails everytime.

Ciao and Thanks.

Giuseppe

Re: ingest csv files containing multi line fields

richgalloway — Thu, 09 Jul 2020 18:19:28 GMT

I've done that in the past and, IIRC, it just worked. Can you share your props?

Re: ingest csv files containing multi line fields

gcusello — Fri, 10 Jul 2020 06:33:58 GMT

Hi @richgalloway ,

thanks for your help.

this is my last props.conf, I used also other tries (e.g. listing fields instead HEADER_FIELD_LINE_NUMBER) but with the same result:

[my_sourcetype] BREAK_ONLY_BEFORE_DATE = DATETIME_CONFIG = CURRENT FIELD_DELIMITER = ; FIELD_QUOTE = " INDEXED_EXTRACTIONS = csv KV_MODE = none LINE_BREAKER = ([\r\n]+) NO_BINARY_CHECK = true SHOULD_LINEMERGE = false category = Structured description = Processes disabled = false pulldown_type = 1 PREAMBLE_REGEX = ^XXX HEADER_FIELD_LINE_NUMBER = 1

Ciao and thanks.

Giuseppe

Re: ingest csv files containing multi line fields

richgalloway — Fri, 10 Jul 2020 13:46:52 GMT

This is what I get when I onboard your data with those settings.

Re: ingest csv files containing multi line fields

gcusello — Fri, 10 Jul 2020 16:02:14 GMT

Hi @richgalloway ,

when I try to ingest my logs I have an event for each row.

Maybe I could use SHOULD_LINEMERGE = true but I don't know how to divide events.

Ciao.

Giuseppe

Re: ingest csv files containing multi line fields

richgalloway — Fri, 10 Jul 2020 16:57:43 GMT

I suspect we're each doing something differently to get the different results, but I'm not sure what that something is.

Using SHOULD_LINEMERGE = true might help, but would break fields that are intended to have multiple lines. Then, as you say, there's the matter of how to break events properly.