How to extract time from preamble data in csv?

[manhalmoussa]

Hello,

i am using UF to ingest a csv file that has the timestamp in preamble data, i would like to extract the timestamp and to remove the preamble data and then ingest the csv. 

the file looks like the table below:

Time stamp	2023-01-26T11:15:00-05:00
info	obj
datainfo	blahblah
datadata	blahblah
field1	field1	field2
value1	1	info1
value2	2	info2
value3	3	info3
value4	4	info4
value5	5	info5
value6	6	info6
value7	7	info7
value8	8	info8
value9	9	info9

my props.conf:

DATETIME_CONFIG =
TIME_PREFIX=Time\sstamp,
MAX_TIMESTAMP_LOOKAHEAD=22
TIME_FORMAT=%Y-%m-%dT%H:%M:%S%z
INDEXED_EXTRACTIONS = CSV
FIELD_HEADER_REGEX = (field1.*)
LINE_BREAKER = ([\r\n]+)
SHOULD_LINEMERGE = false
NO_BINARY_CHECK = true

the issue here is i am able to read the csv and the field names, however the timestamp of the event is the current time and not from the file. how do i fix this?

[scelikok]

Hi @manhalmoussa,

Your MAX_TIMESTAMP_LOOKAHEAD setting seems wrong, please try this.

MAX_TIMESTAMP_LOOKAHEAD=25
TIME_FORMAT=%Y-%m-%dT%H:%M:%S%:z

[manhalmoussa]

Hi scelikok,

i have updated the props to the new settings, unfortunately it doesn't work. i have noticed that the time settings works up untill i add: 

INDEXED_EXTRACTIONS = CSV 

 if i removed the INDEXED_EXTRACTIONS, then i will lose the ability to parse and capture the fields. that means i can only get the props.conf to capture the time correctly but not the fields or vice versa. 

[richgalloway]

The example timestamp doesn't match the format in TIME_FORMAT.  Try 

TIME_FORMAT=%Y-%m-%dT%H:%M:%S%:z

[manhalmoussa]

Thank you for your answer, unfortunately it doesn't work.