topic Re: How to extract time from preamble data in csv? in Getting Data In

How to extract time from preamble data in csv?

manhalmoussa — Fri, 27 Jan 2023 21:29:52 GMT

Hello,

i am using UF to ingest a csv file that has the timestamp in preamble data, i would like to extract the timestamp and to remove the preamble data and then ingest the csv.

the file looks like the table below:

Time stamp	2023-01-26T11:15:00-05:00
info	obj
datainfo	blahblah
datadata	blahblah

field1	field1	field2
value1	1	info1
value2	2	info2
value3	3	info3
value4	4	info4
value5	5	info5
value6	6	info6
value7	7	info7
value8	8	info8
value9	9	info9

my props.conf:

DATETIME_CONFIG =
TIME_PREFIX=Time\sstamp,
MAX_TIMESTAMP_LOOKAHEAD=22
TIME_FORMAT=%Y-%m-%dT%H:%M:%S%z
INDEXED_EXTRACTIONS = CSV
FIELD_HEADER_REGEX = (field1.*)
LINE_BREAKER = ([\r\n]+)
SHOULD_LINEMERGE = false
NO_BINARY_CHECK = true

the issue here is i am able to read the csv and the field names, however the timestamp of the event is the current time and not from the file. how do i fix this?

Re: Extracting Time from preamble data in csv

richgalloway — Fri, 27 Jan 2023 20:45:17 GMT

The example timestamp doesn't match the format in TIME_FORMAT. Try

TIME_FORMAT=%Y-%m-%dT%H:%M:%S%:z

Re: Extracting Time from preamble data in csv

manhalmoussa — Fri, 27 Jan 2023 20:56:30 GMT

Thank you for your answer, unfortunately it doesn't work.

Re: How to extract time from preamble data in csv?

scelikok — Sat, 28 Jan 2023 06:34:42 GMT

Hi @manhalmoussa,

Your MAX_TIMESTAMP_LOOKAHEAD setting seems wrong, please try this.

MAX_TIMESTAMP_LOOKAHEAD=25 TIME_FORMAT=%Y-%m-%dT%H:%M:%S%:z

Re: How to extract time from preamble data in csv?

manhalmoussa — Mon, 30 Jan 2023 16:18:42 GMT

Hi scelikok,

i have updated the props to the new settings, unfortunately it doesn't work. i have noticed that the time settings works up untill i add:

INDEXED_EXTRACTIONS = CSV

if i removed the INDEXED_EXTRACTIONS, then i will lose the ability to parse and capture the fields. that means i can only get the props.conf to capture the time correctly but not the fields or vice versa.