Hello,
i am using UF to ingest a csv file that has the timestamp in preamble data, i would like to extract the timestamp and to remove the preamble data and then ingest the csv.
the file looks like the table below:
Time stamp | 2023-01-26T11:15:00-05:00 | |
info | obj | |
datainfo | blahblah | |
datadata | blahblah | |
field1 | field1 | field2 |
value1 | 1 | info1 |
value2 | 2 | info2 |
value3 | 3 | info3 |
value4 | 4 | info4 |
value5 | 5 | info5 |
value6 | 6 | info6 |
value7 | 7 | info7 |
value8 | 8 | info8 |
value9 | 9 | info9 |
my props.conf:
DATETIME_CONFIG =
TIME_PREFIX=Time\sstamp,
MAX_TIMESTAMP_LOOKAHEAD=22
TIME_FORMAT=%Y-%m-%dT%H:%M:%S%z
INDEXED_EXTRACTIONS = CSV
FIELD_HEADER_REGEX = (field1.*)
LINE_BREAKER = ([\r\n]+)
SHOULD_LINEMERGE = false
NO_BINARY_CHECK = true
the issue here is i am able to read the csv and the field names, however the timestamp of the event is the current time and not from the file. how do i fix this?
Hi @manhalmoussa,
Your MAX_TIMESTAMP_LOOKAHEAD setting seems wrong, please try this.
MAX_TIMESTAMP_LOOKAHEAD=25
TIME_FORMAT=%Y-%m-%dT%H:%M:%S%:z
Hi scelikok,
i have updated the props to the new settings, unfortunately it doesn't work. i have noticed that the time settings works up untill i add:
INDEXED_EXTRACTIONS = CSV
if i removed the INDEXED_EXTRACTIONS, then i will lose the ability to parse and capture the fields. that means i can only get the props.conf to capture the time correctly but not the fields or vice versa.
The example timestamp doesn't match the format in TIME_FORMAT. Try
TIME_FORMAT=%Y-%m-%dT%H:%M:%S%:z
Thank you for your answer, unfortunately it doesn't work.