topic Re: Support CSV files that have no CRLF at end of last line? in Getting Data In

Support CSV files that have no CRLF at end of last line?

wryanthomas — Mon, 16 Dec 2024 17:00:55 GMT

In the TA documentation at https://splunk.github.io/splunk-add-on-for-amazon-web-services/S3/ -- it is stated, "Ensure that all files have a carriage return at the end of each file. Otherwise, the last line of the CSV file will not be indexed." But the CSV standard (https://www.ietf.org/rfc/rfc4180.txt) does not require a CRLF at end of last row. Can you please remedy this so a standard-compliant CSV file without a final CRLF still works and ingests the final row? Some source solutions only output CSV files in this way (without final CRLF).

Re: Support CSV files that have no CRLF at end of last line?

isoutamo — Mon, 16 Dec 2024 20:52:14 GMT

If you want this kind of feature into this TA, you must ask it from splunk support and/or ideas.splunk.com.

Re: Support CSV files that have no CRLF at end of last line?

wryanthomas — Fri, 20 Dec 2024 20:16:42 GMT

Thanks. I have submitted it as an idea at ideas.splunk.com. https://ideas.splunk.com/ideas/APPSID-I-944

Re: Support CSV files that have no CRLF at end of last line?

PickleRick — Fri, 20 Dec 2024 22:04:59 GMT

Mind you this RFC is informational and only aims to document common practices. It's by no means to be a standard.

Re: Support CSV files that have no CRLF at end of last line?

wryanthomas — Fri, 20 Dec 2024 22:16:10 GMT

Thanks. I see that RFC 4108 specifies the same thing -- that the last line could end with or without a final CRLF:

https://www.loc.gov/preservation/digital/formats/fdd/fdd000323.shtml

In the Notes, General section at the end of the document, "The last record in a file may or may not end with a line break character."