topic Issues with TIME_ROMAT Field in props.conf file in Security

Issues with TIME_ROMAT Field in props.conf file

SplunkDash — Mon, 28 Aug 2023 08:40:29 GMT

Hello,

I have some issues with the TIME_FORMAT field in props.conf file, getting some error messages "Failed to parse timestamp, defaulting to file modtime" . My pprops.conf file and a couple of sample events are given below. Any help will be highly appreciated. Thank you!

00000000|REG|USER|LOGIN|rsd56qa|00000000||10.108.125.71|01||2023-05-09T11:00:59.000-04.00||||||success| 00000000|REG|USER|LOGIN|adb23rm|00000000||10.108.125.71|06||2023-05-10T06:05:43.000-04.00||||||success|

[sourcetype] SHOULD_LINEMERGE=false LINE_BREAKER=([\r\n]+) TIME_PREFIX=([^\|]+\|){10} TIME_FORMAT=%Y-%m-%dT%H:%M:%S.%3N MAX_TIMESTAMP_LOOKAHEAD=30 TRUNCATE=2500

Re: Issues with TIME_ROMAT Field in props.conf file

manjunathmeti — Mon, 28 Aug 2023 03:34:21 GMT

@SplunkDash ,

Your time prefix needs to change. Also, don't set MAX_TIMESTAMP_LOOKAHEAD if your timestamp is in the text following the end of the TIME_PREFIX regex match. Try this:

[sourcetype] SHOULD_LINEMERGE=false LINE_BREAKER=([\r\n]+) TIME_PREFIX=.+\|[\d\.]+\|\d+\|\| TIME_FORMAT=%Y-%m-%dT%H:%M:%S.%3N # MAX_TIMESTAMP_LOOKAHEAD=30 TRUNCATE=2500

Re: Issues with TIME_ROMAT Field in props.conf file

SplunkDash — Mon, 28 Aug 2023 03:42:28 GMT

@manjunathmeti ,

Thank you so much for your quick response. But issue with as sometimes we have values between || before timetsamp.

00000000|REG|USER|LOGIN|rsd56qa|00000000||10.108.125.71|01||2023-05-09T11:00:59.000-04.00||||||success|

Re: Issues with TIME_ROMAT Field in props.conf file

SplunkDash — Mon, 28 Aug 2023 08:43:49 GMT

@manjunathmeti ,

Yes, I just tweak a little with my props, and working now as expected. Thank you so much for your support in these efforts.

[sourcetype] SHOULD_LINEMERGE=false LINE_BREAKER=([\r\n]+) TIME_PREFIX=([^\|]*\|){10} # I used "*" instead of "+" TIME_FORMAT=%Y-%m-%dT%H:%M:%S.%3N MAX_TIMESTAMP_LOOKAHEAD=30 TRUNCATE=2500

Re: Issues with TIME_ROMAT Field in props.conf file

isoutamo — Mon, 28 Aug 2023 07:32:13 GMT

as this seems to be "|" separated file, I said that you should/could use that when ingesting this file.

Then you have wrongly defined TZ on your event! You have "-04.00" when it should be "-04:00" or "-0400"! This will break TZ recognition! You should ask that this will fixed or another option is remove ".00" when you use that. You could it by "%:::z" which should handle this, but You should remember that there are couple of TZ where the difference is XX:30 instead of XX:00 and those will be broken until source formatting is fixed!

Just tested and if you have "%:z" and "-04.30" then it just use "-04" and you will get wrong TZ, but with "-04.00" it handle "correctly" as there dropped part is "00".

You could try something like

[<Your sourcetype>] SHOULD_LINEMERGE=true LINE_BREAKER=([\r\n]+) NO_BINARY_CHECK=true FIELD_DELIMITER=| FIELD_NAMES=f1,REG,USER,login,f5,f6,f7,src_ip,f9,f10,ts,f12,f13,f14,f15,f16,status TIME_FORMAT=%Y-%m-%dT%H:%M:%S.%3Q%:z TIME_PREFIX=([^\|]*\|){10} MAX_TIMESTAMP_LOOKAHEAD=29

on your UF. Just fix those FIELD_NAMES to correct ones.

r. Ismo

Re: Issues with TIME_ROMAT Field in props.conf file

SplunkDash — Mon, 28 Aug 2023 13:12:00 GMT

Hello @isoutamo,

Thank you so much for your recommendation. It's working as expected, only chance I needed to make marked as Bold:

[<Your sourcetype>]
SHOULD_LINEMERGE=true
LINE_BREAKER=([\r\n]+)
NO_BINARY_CHECK=true
FIELD_DELIMITER=|
FIELD_NAMES=f1,REG,USER,login,f5,f6,f7,src_ip,f9,f10,ts,f12,f13,f14,f15,f16,status
TIME_FORMAT=%Y-%m-%dT%H:%M:%S.%3Q%:z (-%z)
TIME_PREFIX=([^\|]*\|){10}
MAX_TIMESTAMP_LOOKAHEAD=29

Re: Issues with TIME_ROMAT Field in props.conf file

isoutamo — Mon, 28 Aug 2023 16:41:17 GMT

Good to hear that this is working!
BUT still you have this issue on your TZ definition on log file. If you ever get logs from TZ which has its xx:30 (like some Indian like -05:30) shift instead of full hour, those will get a wrong UTC time on splunk.