Getting Data In

Why is my tsv data out of order

tkwaller_2
Communicator

Hello

We are parsing data from a TSV source

The data file has a header that is very long, about 281 columns.
What is happening is that we are getting data in the wrong fields.
For example:

Field:
data_poc_technical_name

Values:  
todd.waller@toddwaller.com

this should be the value in the field: data_poc_technical_email.
I also notice that when exporting the data its out of order as well, maybe the issue lies in parsing and configs?
Originally when I tested I got field names in the fields values, I think when data was NULL so I removed FIELD_NAMES from the props and it seemed to have parsed correctly but now doesnt look that way.

These are the props on the UF

[fp:tsv]
FIELD_DELIMITER = \t
HEADER_FIELD_DELIMITER = \t
INDEXED_EXTRACTIONS = TSV
TIMESTAMP_FIELDS = md_createdAt

and the props from the indexer

[fp:tsv]
FIELD_DELIMITER = \t
HEADER_FIELD_DELIMITER = \t
TIMESTAMP_FIELDS = md_createdAt
KV_MODE = none

Any thoughts?
Thanks for the help!
Todd

0 Karma
1 Solution

tkwaller_2
Communicator

I figured this out
It seems the issue was timestamping. Once I fixed the timestamp field recognition and reindexed the data it seems to be correct now.

My edited props on the UF were

[fp:tsv]
TIME_FORMAT = %m/%d/%Y %H:%M:%S
FIELD_DELIMITER = \t
HEADER_FIELD_DELIMITER = \t
INDEXED_EXTRACTIONS = TSV
TIMESTAMP_FIELDS = md.createdAt

In case it helps anyone else.

View solution in original post

0 Karma

tkwaller_2
Communicator

I figured this out
It seems the issue was timestamping. Once I fixed the timestamp field recognition and reindexed the data it seems to be correct now.

My edited props on the UF were

[fp:tsv]
TIME_FORMAT = %m/%d/%Y %H:%M:%S
FIELD_DELIMITER = \t
HEADER_FIELD_DELIMITER = \t
INDEXED_EXTRACTIONS = TSV
TIMESTAMP_FIELDS = md.createdAt

In case it helps anyone else.

0 Karma
Get Updates on the Splunk Community!

Aligning Observability Costs with Business Value: Practical Strategies

 Join us for an engaging Tech Talk on Aligning Observability Costs with Business Value: Practical ...

Mastering Data Pipelines: Unlocking Value with Splunk

 In today's AI-driven world, organizations must balance the challenges of managing the explosion of data with ...

Splunk Up Your Game: Why It's Time to Embrace Python 3.9+ and OpenSSL 3.0

Did you know that for Splunk Enterprise 9.4, Python 3.9 is the default interpreter? This shift is not just a ...