Issues with TIME_ROMAT Field in props.conf file

[SplunkDash]

Hello,

I have some issues with the TIME_FORMAT field in props.conf file, getting some error messages "Failed to parse timestamp, defaulting to file modtime" . My pprops.conf file and a couple of sample events are given below. Any help will be highly appreciated. Thank you!

00000000|REG|USER|LOGIN|rsd56qa|00000000||10.108.125.71|01||2023-05-09T11:00:59.000-04.00||||||success|

00000000|REG|USER|LOGIN|adb23rm|00000000||10.108.125.71|06||2023-05-10T06:05:43.000-04.00||||||success|

[sourcetype]
SHOULD_LINEMERGE=false
LINE_BREAKER=([\r\n]+)
TIME_PREFIX=([^\|]+\|){10}
TIME_FORMAT=%Y-%m-%dT%H:%M:%S.%3N
MAX_TIMESTAMP_LOOKAHEAD=30
TRUNCATE=2500

[manjunathmeti]

@SplunkDash ,

Your time prefix needs to change. Also, don't set MAX_TIMESTAMP_LOOKAHEAD if your timestamp is in the text following the end of the TIME_PREFIX regex match. Try this:

[sourcetype]
SHOULD_LINEMERGE=false
LINE_BREAKER=([\r\n]+)
TIME_PREFIX=.+\|[\d\.]+\|\d+\|\|
TIME_FORMAT=%Y-%m-%dT%H:%M:%S.%3N
# MAX_TIMESTAMP_LOOKAHEAD=30
TRUNCATE=2500

[SplunkDash]

@manjunathmeti ,

Thank you so much for your quick response. But issue with as sometimes we have values between || before timetsamp. 

00000000|REG|USER|LOGIN|rsd56qa|00000000||10.108.125.71|01||2023-05-09T11:00:59.000-04.00||||||success|

[SplunkDash]

@manjunathmeti ,

Yes, I just tweak a little with my props, and working now as expected. Thank you so much for your support in these efforts.

[sourcetype]
SHOULD_LINEMERGE=false
LINE_BREAKER=([\r\n]+)

TIME_PREFIX=([^\|]*\|){10}    # I used "*" instead of "+"

TIME_FORMAT=%Y-%m-%dT%H:%M:%S.%3N
MAX_TIMESTAMP_LOOKAHEAD=30
TRUNCATE=2500

[isoutamo · Accepted solution]

Hi

as this seems to be "|" separated file, I said that you should/could use that when ingesting this file.

Then you have wrongly defined TZ on your event! You have "-04.00" when it should be "-04:00" or "-0400"! This will break TZ recognition! You should ask that this will fixed or another option is remove ".00" when you use that. You could it by "%:::z" which should handle this, but You should remember that there are couple of TZ where the difference is XX:30 instead of XX:00 and those will be broken until source formatting is fixed!

Just tested and if you have "%:z" and "-04.30" then it just use "-04" and you will get wrong TZ, but with "-04.00" it handle "correctly" as there dropped part is "00".

You could try something like 

[<Your sourcetype>]
SHOULD_LINEMERGE=true
LINE_BREAKER=([\r\n]+)
NO_BINARY_CHECK=true
FIELD_DELIMITER=|
FIELD_NAMES=f1,REG,USER,login,f5,f6,f7,src_ip,f9,f10,ts,f12,f13,f14,f15,f16,status
TIME_FORMAT=%Y-%m-%dT%H:%M:%S.%3Q%:z
TIME_PREFIX=([^\|]*\|){10}
MAX_TIMESTAMP_LOOKAHEAD=29

on your UF.  Just fix those FIELD_NAMES to correct ones.

r. Ismo

[SplunkDash]

Hello @isoutamo,

Thank you so much for your recommendation. It's working as expected, only chance I needed to make marked as Bold:

[<Your sourcetype>]
SHOULD_LINEMERGE=true
LINE_BREAKER=([\r\n]+)
NO_BINARY_CHECK=true
FIELD_DELIMITER=|
FIELD_NAMES=f1,REG,USER,login,f5,f6,f7,src_ip,f9,f10,ts,f12,f13,f14,f15,f16,status
TIME_FORMAT=%Y-%m-%dT%H:%M:%S.%3Q%:z (-%z)
TIME_PREFIX=([^\|]*\|){10}
MAX_TIMESTAMP_LOOKAHEAD=29

[isoutamo]

Good to hear that this is working!
BUT still you have this issue on your TZ definition on log file. If you ever get logs from TZ which has its xx:30 (like some Indian like -05:30) shift instead of full hour, those will get a wrong UTC time on splunk.