CSV file input issue: does not correctly extract at index time a field bounded by 2 double quotes

[tomasmijares]

I have defined the following sourcetype for a CSV file data input without headers:

[test_csv]
SHOULD_LINEMERGE = false
TRANSFORMS = drop_start_and_interim
INDEXED_EXTRACTIONS = csv
FIELD_NAMES = 1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95,96,97,98,99,100,101,102,103,104,105,106,107,108,109,110,111,112,113,114,115,116,117,118,119,120,121,122,123,124,125,126,127,128,129,130,131,132,133,134,135,136,137,138,139,140,141,142,143,144,145,146,147,148,149,150,151,152,153,154,155,156,157,158,159,160,161,162,163,164,165,166,167,168,169,170,171,172,173,174,175,176,177,178,179,180,181,182,183,184,185,186,187
KV_MODE = none
LINE_BREAKER = ([\r\n]+)
NO_BINARY_CHECK = true
TIMESTAMP_FIELDS = 14
category = Structured
description = Comma-separated value format. Set header and other settings in "Delimited Settings"
disabled = false
pulldown_type = true

When I index a test file I see that there is one of the destination fields that is not correctly extracted, this field is bounded by 2 double quotes and is extacted together with the next field as a single field.

A sample raw with the problem is the following where I have marked the field in red:

2,"127.0.0.1",5060,"258670334_106281015@83.72.181.1","258670334_106281015@83.72.181.1","258670334_106281015@83.72.181.1","SIP",,,"<sip:+34765300391@83.72.181.1;user=phone>;tag=gK0a655dd7","<sip:+376826792@193.178.74.21;user=phone>",1,1611,"14:35:43.412 CET Jan 09 2024","14:35:52.884 CET Jan 09 2024","15:02:43.220 CET Jan 09 2024",1,"s0p2",53,"s0p0",52,"IMS","IX","localhost:154311320","PCMA","IX","83.72.181.97",40072,"193.178.74.21",20526,"IMS","10.12.162.20",16864,"10.12.45.10",25732,0,0,0,0,0,0,0,1,17551834,80513,9284,440,"localhost:154311321","PCMA","IMS","10.12.45.10",25732,"10.12.162.20",16864,"IX","193.178.74.21",20526,"83.72.181.97",40072,0,0,0,0,0,0,0,2,17552488,80516,9284,440,,,,"0.0.0.0",0,"0.0.0.0",0,,"0.0.0.0",0,"0.0.0.0",0,0,0,0,0,0,0,0,0,0,0,,,,"0.0.0.0",0,"0.0.0.0",0,,"0.0.0.0",0,"0.0.0.0",0,0,0,0,0,0,0,0,0,0,0,"bb6c6d3001911f060e83641d9e64",""aaa://inf.tsa"","SCZ9.0.0 Patch 2 (Build 211)","GMT-01:00",245,"sip:+376826792@193.178.74.21:5060;user=phone",,,,,"sip:+34765300391@83.72.181.1:5060;user=phone","193.178.74.21:5060","83.72.181.1:5060","10.12.193.4:5060","10.59.90.201:5060",,3,2,0,0,"sip:+376826792@FO01-vICSCF-01.ims.mnc006.mcc333.3gppnetwork.org:5060;user=phone",,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,"15:02:43.220 CET Jan 09 2024","15:02:43.220 CET Jan 09 2024","00:00:00.000 UTC Jan 01 1970","00:00:00.000 UTC Jan 01 1970","audio","audio",,,17551834,80513,17552052,80514,0,0,0,0,19516010

The content of the field 117 is:

"aaa://inf.tsa","SCZ9.0.0 Patch 2 (Build 211)

It corresponds to the fields 117 and 118 concatenated, and the following fields are all offset one position

I have tried to replace the 2 double quotes by 1 in 2 ways:

1.  Adding the line SEDCMD = s/""/"/g at the first line in the sourcetype definition in the props.conf but it only changes the _raw and still have the same issue extracting the field 117 and the offset of the following fields
2. I have tried to overwrite de _raw replacing the 2 double quotes by 1 with the following transforms:

[rewrite_raw]
INGEST_EVAL = _raw:=replace(_raw, "\"\"", "\"")

Applied in the sourcetype after the other transform that drops some kind of rows based on the value of the first field

TRANSFORMS = drop_start_and_interim, rewrite_raw

And the result is the same, the _raw is changed but the issue extracting the filed 117 and offset of the followings persists

I also have tried to rewrite the _raw with the following transform and it neither has solved the problem, the result has been the same:

[remove_double_quotes]
SOURCE_KEY = _raw
REGEX = (?:\""|\"|)(.*?)(?:\"\"|\"|)\,(?:\""|\"|)(.*?)(?:\"\"|\"|)\,(?:\""|\"|)(.*?)(?:\"\"|\"|)\,(?:\""|\"|)(.*?)(?:\"\"|\"|)\,(?:\""|\"|)(.*?)(?:\"\"|\"|)\,(?:\""|\"|)(.*?)(?:\"\"|\"|)\,(?:\""|\"|)(.*?)(?:\"\"|\"|)\,(?:\""|\"|)(.*?)(?:\"\"|\"|)\,(?:\""|\"|)(.*?)(?:\"\"|\"|)\,(?:\""|\"|)(.*?)(?:\"\"|\"|)\,(?:\""|\"|)(.*?)(?:\"\"|\"|)\,(?:\""|\"|)(.*?)(?:\"\"|\"|)\,(?:\""|\"|)(.*?)(?:\"\"|\"|)\,(?:\""|\"|)(.*?)(?:\"\"|\"|)\,(?:\""|\"|)(.*?)(?:\"\"|\"|)\,(?:\""|\"|)(.*?)(?:\"\"|\"|)\,(?:\""|\"|)(.*?)(?:\"\"|\"|)\,(?:\""|\"|)(.*?)(?:\"\"|\"|)\,(?:\""|\"|)(.*?)(?:\"\"|\"|)\,(?:\""|\"|)(.*?)(?:\"\"|\"|)\,(?:\""|\"|)(.*?)(?:\"\"|\"|)\,(?:\""|\"|)(.*?)(?:\"\"|\"|)\,(?:\""|\"|)(.*?)(?:\"\"|\"|)\,(?:\""|\"|)(.*?)(?:\"\"|\"|)\,(?:\""|\"|)(.*?)(?:\"\"|\"|)\,(?:\""|\"|)(.*?)(?:\"\"|\"|)\,(?:\""|\"|)(.*?)(?:\"\"|\"|)\,(?:\""|\"|)(.*?)(?:\"\"|\"|)\,(?:\""|\"|)(.*?)(?:\"\"|\"|)\,(?:\""|\"|)(.*?)(?:\"\"|\"|)\,(?:\""|\"|)(.*?)(?:\"\"|\"|)\,(?:\""|\"|)(.*?)(?:\"\"|\"|)\,(?:\""|\"|)(.*?)(?:\"\"|\"|)\,(?:\""|\"|)(.*?)(?:\"\"|\"|)\,(?:\""|\"|)(.*?)(?:\"\"|\"|)\,(?:\""|\"|)(.*?)(?:\"\"|\"|)\,(?:\""|\"|)(.*?)(?:\"\"|\"|)\,(?:\""|\"|)(.*?)(?:\"\"|\"|)\,(?:\""|\"|)(.*?)(?:\"\"|\"|)\,(?:\""|\"|)(.*?)(?:\"\"|\"|)\,(?:\""|\"|)(.*?)(?:\"\"|\"|)\,(?:\""|\"|)(.*?)(?:\"\"|\"|)\,(?:\""|\"|)(.*?)(?:\"\"|\"|)\,(?:\""|\"|)(.*?)(?:\"\"|\"|)\,(?:\""|\"|)(.*?)(?:\"\"|\"|)\,(?:\""|\"|)(.*?)(?:\"\"|\"|)\,(?:\""|\"|)(.*?)(?:\"\"|\"|)\,(?:\""|\"|)(.*?)(?:\"\"|\"|)\,(?:\""|\"|)(.*?)(?:\"\"|\"|)\,(?:\""|\"|)(.*?)(?:\"\"|\"|)\,(?:\""|\"|)(.*?)(?:\"\"|\"|)\,(?:\""|\"|)(.*?)(?:\"\"|\"|)\,(?:\""|\"|)(.*?)(?:\"\"|\"|)\,(?:\""|\"|)(.*?)(?:\"\"|\"|)\,(?:\""|\"|)(.*?)(?:\"\"|\"|)\,(?:\""|\"|)(.*?)(?:\"\"|\"|)\,(?:\""|\"|)(.*?)(?:\"\"|\"|)\,(?:\""|\"|)(.*?)(?:\"\"|\"|)\,(?:\""|\"|)(.*?)(?:\"\"|\"|)\,(?:\""|\"|)(.*?)(?:\"\"|\"|)\,(?:\""|\"|)(.*?)(?:\"\"|\"|)\,(?:\""|\"|)(.*?)(?:\"\"|\"|)\,(?:\""|\"|)(.*?)(?:\"\"|\"|)\,(?:\""|\"|)(.*?)(?:\"\"|\"|)\,(?:\""|\"|)(.*?)(?:\"\"|\"|)\,(?:\""|\"|)(.*?)(?:\"\"|\"|)\,(?:\""|\"|)(.*?)(?:\"\"|\"|)\,(?:\""|\"|)(.*?)(?:\"\"|\"|)\,(?:\""|\"|)(.*?)(?:\"\"|\"|)\,(?:\""|\"|)(.*?)(?:\"\"|\"|)\,(?:\""|\"|)(.*?)(?:\"\"|\"|)\,(?:\""|\"|)(.*?)(?:\"\"|\"|)\,(?:\""|\"|)(.*?)(?:\"\"|\"|)\,(?:\""|\"|)(.*?)(?:\"\"|\"|)\,(?:\""|\"|)(.*?)(?:\"\"|\"|)\,(?:\""|\"|)(.*?)(?:\"\"|\"|)\,(?:\""|\"|)(.*?)(?:\"\"|\"|)\,(?:\""|\"|)(.*?)(?:\"\"|\"|)\,(?:\""|\"|)(.*?)(?:\"\"|\"|)\,(?:\""|\"|)(.*?)(?:\"\"|\"|)\,(?:\""|\"|)(.*?)(?:\"\"|\"|)\,(?:\""|\"|)(.*?)(?:\"\"|\"|)\,(?:\""|\"|)(.*?)(?:\"\"|\"|)\,(?:\""|\"|)(.*?)(?:\"\"|\"|)\,(?:\""|\"|)(.*?)(?:\"\"|\"|)\,(?:\""|\"|)(.*?)(?:\"\"|\"|)\,(?:\""|\"|)(.*?)(?:\"\"|\"|)\,(?:\""|\"|)(.*?)(?:\"\"|\"|)\,(?:\""|\"|)(.*?)(?:\"\"|\"|)\,(?:\""|\"|)(.*?)(?:\"\"|\"|)\,(?:\""|\"|)(.*?)(?:\"\"|\"|)\,(?:\""|\"|)(.*?)(?:\"\"|\"|)\,(?:\""|\"|)(.*?)(?:\"\"|\"|)\,(?:\""|\"|)(.*?)(?:\"\"|\"|)\,(?:\""|\"|)(.*?)(?:\"\"|\"|)\,(?:\""|\"|)(.*?)(?:\"\"|\"|)\,(?:\""|\"|)(.*?)(?:\"\"|\"|)\,(?:\""|\"|)(.*?)(?:\"\"|\"|)\,(?:\""|\"|)(.*?)(?:\"\"|\"|)\,(?:\""|\"|)(.*?)(?:\"\"|\"|)\,(?:\""|\"|)(.*?)(?:\"\"|\"|)\,(?:\""|\"|)(.*?)(?:\"\"|\"|)\,(?:\""|\"|)(.*?)(?:\"\"|\"|)\,(?:\""|\"|)(.*?)(?:\"\"|\"|)\,(?:\""|\"|)(.*?)(?:\"\"|\"|)\,(?:\""|\"|)(.*?)(?:\"\"|\"|)\,(?:\""|\"|)(.*?)(?:\"\"|\"|)\,(?:\""|\"|)(.*?)(?:\"\"|\"|)\,(?:\""|\"|)(.*?)(?:\"\"|\"|)\,(?:\""|\"|)(.*?)(?:\"\"|\"|)\,(?:\""|\"|)(.*?)(?:\"\"|\"|)\,(?:\""|\"|)(.*?)(?:\"\"|\"|)\,(?:\""|\"|)(.*?)(?:\"\"|\"|)\,(?:\""|\"|)(.*?)(?:\"\"|\"|)\,(?:\""|\"|)(.*?)(?:\"\"|\"|)\,(?:\""|\"|)(.*?)(?:\"\"|\"|)\,(?:\""|\"|)(.*?)(?:\"\"|\"|)\,(?:\""|\"|)(.*?)(?:\"\"|\"|)\,(?:\""|\"|)(.*?)(?:\"\"|\"|)\,(?:\""|\"|)(.*?)(?:\"\"|\"|)\,(?:\""|\"|)(.*?)(?:\"\"|\"|)\,(?:\""|\"|)(.*?)(?:\"\"|\"|)\,(?:\""|\"|)(.*?)(?:\"\"|\"|)\,(?:\""|\"|)(.*?)(?:\"\"|\"|)\,(?:\""|\"|)(.*?)(?:\"\"|\"|)\,(?:\""|\"|)(.*?)(?:\"\"|\"|)\,(?:\""|\"|)(.*?)(?:\"\"|\"|)\,(?:\""|\"|)(.*?)(?:\"\"|\"|)\,(?:\""|\"|)(.*?)(?:\"\"|\"|)\,(?:\""|\"|)(.*?)(?:\"\"|\"|)\,(?:\""|\"|)(.*?)(?:\"\"|\"|)\,(?:\""|\"|)(.*?)(?:\"\"|\"|)\,(?:\""|\"|)(.*?)(?:\"\"|\"|)\,(?:\""|\"|)(.*?)(?:\"\"|\"|)\,(?:\""|\"|)(.*?)(?:\"\"|\"|)\,(?:\""|\"|)(.*?)(?:\"\"|\"|)\,(?:\""|\"|)(.*?)(?:\"\"|\"|)\,(?:\""|\"|)(.*?)(?:\"\"|\"|)\,(?:\""|\"|)(.*?)(?:\"\"|\"|)\,(?:\""|\"|)(.*?)(?:\"\"|\"|)\,(?:\""|\"|)(.*?)(?:\"\"|\"|)\,(?:\""|\"|)(.*?)(?:\"\"|\"|)\,(?:\""|\"|)(.*?)(?:\"\"|\"|)\,(?:\""|\"|)(.*?)(?:\"\"|\"|)\,(?:\""|\"|)(.*?)(?:\"\"|\"|)\,(?:\""|\"|)(.*?)(?:\"\"|\"|)\,(?:\""|\"|)(.*?)(?:\"\"|\"|)\,(?:\""|\"|)(.*?)(?:\"\"|\"|)\,(?:\""|\"|)(.*?)(?:\"\"|\"|)\,(?:\""|\"|)(.*?)(?:\"\"|\"|)\,(?:\""|\"|)(.*?)(?:\"\"|\"|)\,(?:\""|\"|)(.*?)(?:\"\"|\"|)\,(?:\""|\"|)(.*?)(?:\"\"|\"|)\,(?:\""|\"|)(.*?)(?:\"\"|\"|)\,(?:\""|\"|)(.*?)(?:\"\"|\"|)\,(?:\""|\"|)(.*?)(?:\"\"|\"|)\,(?:\""|\"|)(.*?)(?:\"\"|\"|)\,(?:\""|\"|)(.*?)(?:\"\"|\"|)\,(?:\""|\"|)(.*?)(?:\"\"|\"|)\,(?:\""|\"|)(.*?)(?:\"\"|\"|)\,(?:\""|\"|)(.*?)(?:\"\"|\"|)\,(?:\""|\"|)(.*?)(?:\"\"|\"|)\,(?:\""|\"|)(.*?)(?:\"\"|\"|)\,(?:\""|\"|)(.*?)(?:\"\"|\"|)\,(?:\""|\"|)(.*?)(?:\"\"|\"|)\,(?:\""|\"|)(.*?)(?:\"\"|\"|)\,(?:\""|\"|)(.*?)(?:\"\"|\"|)\,(?:\""|\"|)(.*?)(?:\"\"|\"|)\,(?:\""|\"|)(.*?)(?:\"\"|\"|)\,(?:\""|\"|)(.*?)(?:\"\"|\"|)\,(?:\""|\"|)(.*?)(?:\"\"|\"|)\,(?:\""|\"|)(.*?)(?:\"\"|\"|)\,(?:\""|\"|)(.*?)(?:\"\"|\"|)\,(?:\""|\"|)(.*?)(?:\"\"|\"|)\,(?:\""|\"|)(.*?)(?:\"\"|\"|)\,(?:\""|\"|)(.*?)(?:\"\"|\"|)\,(?:\""|\"|)(.*?)(?:\"\"|\"|)\,(?:\""|\"|)(.*?)(?:\"\"|\"|)\,(?:\""|\"|)(.*?)(?:\"\"|\"|)\,(?:\""|\"|)(.*?)(?:\"\"|\"|)\,(?:\""|\"|)(.*?)(?:\"\"|\"|)\,(?:\""|\"|)(.*?)(?:\"\"|\"|)\,(?:\""|\"|)(.*?)(?:\"\"|\"|)\,(?:\""|\"|)(.*?)(?:\"\"|\"|)\,(?:\""|\"|)(.*?)(?:\"\"|\"|)\,(?:\""|\"|)(.*?)(?:\"\"|\"|)\,(?:\""|\"|)(.*)(?:\"\"|\"|)
FORMAT = "$1","$2","$3","$4","$5","$6","$7","$8","$9","$10","$11","$12","$13","$14","$15","$16","$17","$18","$19","$20","$21","$22","$23","$24","$25","$26","$27","$28","$29","$30","$31","$32","$33","$34","$35","$36","$37","$38","$39","$40","$41","$42","$43","$44","$45","$46","$47","$48","$49","$50","$51","$52","$53","$54","$55","$56","$57","$58","$59","$60","$61","$62","$63","$64","$65","$66","$67","$68","$69","$70","$71","$72","$73","$74","$75","$76","$77","$78","$79","$80","$81","$82","$83","$84","$85","$86","$87","$88","$89","$90","$91","$92","$93","$94","$95","$96","$97","$98","$99","$100","$101","$102","$103","$104","$105","$106","$107","$108","$109","$110","$111","$112","$113","$114","$115","$116","$117","$118","$119","$120","$121","$122","$123","$124","$125","$126","$127","$128","$129","$130","$131","$132","$133","$134","$135","$136","$137","$138","$139","$140","$141","$142","$143","$144","$145","$146","$147","$148","$149","$150","$151","$152","$153","$154","$155","$156","$157","$158","$159","$160","$161","$162","$163","$164","$165","$166","$167","$168","$169","$170","$171","$172","$173","$174","$175","$176","$177","$178","$179","$180","$181","$182","$183","$184","$185","$186","$187"
DEST_KEY =_raw

Is there any way to solve this problem?

Thank you