topic How do I edit my props.conf for proper line breaking when indexing a CSV file with a large amount of quotes and newlines? in Getting Data In

How do I edit my props.conf for proper line breaking when indexing a CSV file with a large amount of quotes and newlines?

wardallen — Mon, 27 Jul 2015 22:18:00 GMT

I have a csv file that's giving me a headache while trying to index it.
It has 100+ columns, several of which are making life difficult by containing large amounts of things like quotes and newlines.

A sanitised example showing the header line and a problem event:

field1,field2,field3,field4,field5,field6
"55634","Barney","","this field behaves well","","1436504081000"
"","Fred","","Here, have some data

that will make your life very difficult

""should"" you try to parse this puppy","F6E25B","1435307738000"

(The quotes around should are intentional, there's sections of the data that look exactly like that)

I've tried using the following props, to no avail - Barney does the right thing, but Fred's line breaking goes wrong. Can someone point out where I'm going wrong?

BREAK_ONLY_AFTER=\"$                                        
HEADER_FIELD_LINE_NUMBER=1
NO_BINARY_CHECK=true
SHOULD_LINEMERGE=true
TIMESTAMP_FIELDS=field6

This file is created completely new at a regular interval - it's a scheduled database dump. I want to index the entirety each time.
I want to keep the inputs.conf as simple as possible, only defining host, sourcetype and destination index. A parsing app on the indexer will have the props.conf.

Thanks in advance for any help.

Re: How do I edit my props.conf for proper line breaking when indexing a CSV file with a large amount of quotes and newlines?

bmacias84 — Tue, 29 Sep 2020 06:49:04 GMT

Ok, to me the only thing that seems to be consistent in your two examples is every event ends with time (epoch). I would do is use MUST_BREAK_AFTER instead of BREAK_ONLY_AFTER.

[yourSourcetype]
MUST_BREAK_AFTER=,"\d{13}"
HEADER_FIELD_LINE_NUMBER=1
NO_BINARY_CHECK=true
SHOULD_LINEMERGE=true
TIMESTAMP_FIELDS=field6

Re: How do I edit my props.conf for proper line breaking when indexing a CSV file with a large amount of quotes and newlines?

wardallen — Mon, 27 Jul 2015 22:40:39 GMT

Sorry, I think I've given you the wrong idea with my fictional data. The actual data's last column may or may not have a value in it. I'll edit my example data when I can.