Importing a CSV and line breaking within a field

diesel6e — Mon, 28 Sep 2020 15:09:41 GMT

Hi Ninjas-
I am trying to import a csv that is in the following format, with a header that defines the fields-

There is no timestamp field, its not required.

I am able to import the CSV, and event breaking is working fine for the header, and a few of the events.

However, some of the description fields have multiple line breaks e.g.

"Description of event blah...

... blah...

blah..
blah.."

However they are all contained within the " "'s within the | delimeters.

I have the following in props.conf (in the import wizard,without spaces after the _ )

[props]

sourcetype = csvimport

SHOULD_ LINEMERGE= true

CHECK_ FOR_ HEADER = TRUE

FIELD_ DELIMITER= |

FIELD_ QUOTE = "

MUST_BREAK_AFTER = $

I have tried using the below two statements to try and bypass the event breaking, perhaps the regex is not working?

MUST_ NOT_ BREAK_ AFTER = |"

MUST_ NOT_ BREAK_ BEFORE = "|

Is there a better way to do this?

Thanks

Re: Importing a CSV and line breaking within a field

richgalloway — Thu, 31 Oct 2013 17:57:01 GMT

We've done that using a REGEX statement in our transforms.conf.

[csv-transform]
REGEX = (?<fieldA>.*?) | (?<fieldB>.*?) | (?<fieldC>.*?) | (?<fieldD>.*?) | \"(?<description>.*?)\" | (?<fieldF>.*?)

You must also reference this transform in your props.conf. Ours looks like this.

[csv-breaktest]
BREAK_ONLY_BEFORE = (regex unique to our first field)
CHECK_FOR_HEADER = false
KV_MODE = multi
MAX_TIMESTAMP_LOOKAHEAD = 50
NO_BINARY_CHECK = true
SHOULD_LINEMERGE = true
REPORT-csv = csv-transform

topic Re: Importing a CSV and line breaking within a field in Getting Data In

Importing a CSV and line breaking within a field

Re: Importing a CSV and line breaking within a field