Hello
We are parsing data from a TSV source
The data file has a header that is very long, about 281 columns.
What is happening is that we are getting data in the wrong fields.
For example:
Field:
data_poc_technical_name
Values:
todd.waller@toddwaller.com
this should be the value in the field: data_poc_technical_email.
I also notice that when exporting the data its out of order as well, maybe the issue lies in parsing and configs?
Originally when I tested I got field names in the fields values, I think when data was NULL so I removed FIELD_NAMES from the props and it seemed to have parsed correctly but now doesnt look that way.
These are the props on the UF
[fp:tsv]
FIELD_DELIMITER = \t
HEADER_FIELD_DELIMITER = \t
INDEXED_EXTRACTIONS = TSV
TIMESTAMP_FIELDS = md_createdAt
and the props from the indexer
[fp:tsv]
FIELD_DELIMITER = \t
HEADER_FIELD_DELIMITER = \t
TIMESTAMP_FIELDS = md_createdAt
KV_MODE = none
Any thoughts?
Thanks for the help!
Todd
I figured this out
It seems the issue was timestamping. Once I fixed the timestamp field recognition and reindexed the data it seems to be correct now.
My edited props on the UF were
[fp:tsv]
TIME_FORMAT = %m/%d/%Y %H:%M:%S
FIELD_DELIMITER = \t
HEADER_FIELD_DELIMITER = \t
INDEXED_EXTRACTIONS = TSV
TIMESTAMP_FIELDS = md.createdAt
In case it helps anyone else.
I figured this out
It seems the issue was timestamping. Once I fixed the timestamp field recognition and reindexed the data it seems to be correct now.
My edited props on the UF were
[fp:tsv]
TIME_FORMAT = %m/%d/%Y %H:%M:%S
FIELD_DELIMITER = \t
HEADER_FIELD_DELIMITER = \t
INDEXED_EXTRACTIONS = TSV
TIMESTAMP_FIELDS = md.createdAt
In case it helps anyone else.