How do I edit my props.conf for proper line breaking when indexing a CSV file with a large amount of quotes and newlines?

[wardallen]

I have a csv file that's giving me a headache while trying to index it.
It has 100+ columns, several of which are making life difficult by containing large amounts of things like quotes and newlines.

A sanitised example showing the header line and a problem event:

field1,field2,field3,field4,field5,field6
"55634","Barney","","this field behaves well","","1436504081000"
"","Fred","","Here, have some data

that will make your life very difficult

""should"" you try to parse this puppy","F6E25B","1435307738000"

---

(The quotes around should are intentional, there's sections of the data that look exactly like that)

I've tried using the following props, to no avail - Barney does the right thing, but Fred's line breaking goes wrong. Can someone point out where I'm going wrong?

BREAK_ONLY_AFTER=\"$                                        
HEADER_FIELD_LINE_NUMBER=1
NO_BINARY_CHECK=true
SHOULD_LINEMERGE=true
TIMESTAMP_FIELDS=field6

This file is created completely new at a regular interval - it's a scheduled database dump. I want to index the entirety each time.
I want to keep the inputs.conf as simple as possible, only defining host, sourcetype and destination index. A parsing app on the indexer will have the props.conf.

Thanks in advance for any help.

[bmacias84]

Ok, to me the only thing that seems to be consistent in your two examples is every event ends with time (epoch). I would do is use MUST_BREAK_AFTER instead of BREAK_ONLY_AFTER.

[yourSourcetype]
MUST_BREAK_AFTER=,"\d{13}"
HEADER_FIELD_LINE_NUMBER=1
NO_BINARY_CHECK=true
SHOULD_LINEMERGE=true
TIMESTAMP_FIELDS=field6

[wardallen]

Sorry, I think I've given you the wrong idea with my fictional data. The actual data's last column may or may not have a value in it. I'll edit my example data when I can.