topic How to properly parse a CSV file with embedded double quotes on the end of a field before the file is indexed? in Splunk Search

How to properly parse a CSV file with embedded double quotes on the end of a field before the file is indexed?

jhuysing — Mon, 14 Dec 2015 23:45:43 GMT

The field ends with a protected quote followed by another quote

Ex:

 "field1",field2", "field3-sdasds\"textdata blah blah\"", "field4-#$%232",

The embedded quotes are protected, but when the files are processed, it doesn't split the fields correctly and field 3 and 4 end up together.

I have experimented with adding a space between the protected quote and field terminating quote and it seems to work.

field1",field2", "field3-sdasds\"textdata blah blah\" ", "field4-#$%232"

Is there someway to do this automatically before the files are indexed?

Re: How to properly parse a CSV file with embedded double quotes on the end of a field before the file is indexed?

andrew207 — Mon, 14 Dec 2015 23:53:28 GMT

You're gonna have to escape the rogue quote.
field1",field2", "field3-sdasds"textdata blah blah\" ", "field4-#$%232"
Any quote that's supposed to be ingested as data rather than a delimiter should be escaped by whatever software is constructing the logs.

Re: How to properly parse a CSV file with embedded double quotes on the end of a field before the file is indexed?

jhuysing — Tue, 15 Dec 2015 00:04:35 GMT

field3 should look like this "field3-sdasds\"textdata blah blah\"", "field4-#$%232"

Re: How to properly parse a CSV file with embedded double quotes on the end of a field before the file is indexed?

jhuysing — Tue, 15 Dec 2015 00:08:37 GMT

try this again

field3 should look like this "field3-sdasds\"textdata blah blah\"", "field4-#$%232"

Re: How to properly parse a CSV file with embedded double quotes on the end of a field before the file is indexed?

jhuysing — Tue, 15 Dec 2015 00:09:28 GMT

ok how do enter backslashes here so they don't get absorbed

Re: How to properly parse a CSV file with embedded double quotes on the end of a field before the file is indexed?

ppablo — Tue, 15 Dec 2015 01:27:08 GMT

Hi @jhuysing

To get backslashes to render properly, you have to wrap your line of text in back ticks like this so lines like \backslash\backslash\ \ \ will show up as expected. If you're every sharing a .conf stanza, it's best to highlight the entire block and click on the "Code Sample" button in the text editing tools above the text box, especially when showing anything with regular expressions. For example:

[stanza]
REGEX = *\<&>\*