DateParserverbose warning logs despite having proper Time format and prefix in props.conf

[likithgowda]

Hey community,

Need your help!!!!

We have lot of internal warn logs for DateParserverbose issue in our splunk prod environment despite passing correct values in TIME_FORMAT, TIME_PREFIX and MAX_TIMESTAMP_LOOKAHEAD attributes in our props.conf. I have listed down warn logs, sample logs and props.conf for your reference.

e.g internal warn log- Failed to parsetimestamp in first MAX_TIMESTAMP_LOOKAHEAD (30) characters of event. Defaulting to timestamp of previous event for sourcetype-test

Sample raw event logs:
Mar 1 07:31:00 xxxxxxx info-message(time=2023-03-01T07:31:00.137, appname=abc, user=john, server=xxx, port=123, msg=logged in) [] [logger] [https:xxxx]
Mar 1 08:29:33 xxxxxxx info-message(time=2023-03-01T08:29:33.135, appname=abc, user=moon, server=yyy, port=897, msg=logged in) [] [logger] [https:xxxx]

Below is our props and transforms that is used to ingest only clean & required logs to splunk prod:

[sourcetype-test]
SHOULD_LINEMERGE = false
LINE_BREAKER = (time\=)|\w+\s+\d+\s+\d+:\d+:\d+|\)
TIME_PREFIX = ^
TIME_FORMAT = %Y-%m-%dT%H:%M:%S.%3QZ
MAX_TIMESTAMP_LOOKAHEAD = 32
TRANSFORMS-test = test_null, test_parsing

[test_null]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[test_parsing]
REGEX = appname
DEST_KEY = queue
FORMAT = indexQueue

Below are the clean log samples that are ingested to splunk as expected but when I check for internal logs for this sourcetype I am seeing lot of warnings for DateParserverbose. So, just wanted to know 1) why there are warn logs when time related settings are correct and is there any way out to fix my props configs to avoid warn logs related to DateParserverbose ?

time=2023-03-01T07:31:00.137, appname=abc, user=john, server=xxx, port=123, msg=logged in
time=2023-03-01T08:29:33.135, appname=abc, user=moon, server=yyy, port=897, msg=logged in

Thanks in advance!!

[likithgowda]

Updated raw event logs :

Mar 1 07:31:00 xxxxxxx info-message(time=2023-03-01T07:31:00.137Z, appname=abc, user=john, server=xxx, port=123, msg=logged in) [] [logger] [https:xxxx]
Mar 1 08:29:33 xxxxxxx info-message(time=2023-03-01T08:29:33.135Z, appname=abc, user=moon, server=yyy, port=897, msg=logged in) [] [logger] [https:xxxx]

[yuanliu]

Maybe because TIME_FORMAT is improper?  Try without Z

TIME_FORMAT = %Y-%m-%dT%H:%M:%S.%3Q

You can test these two using the example you gave above.

| makeresults
| eval test=split("Mar 1 07:31:00 xxxxxxx info-message(time=2023-03-01T07:31:00.137, appname=abc, user=john, server=xxx, port=123, msg=logged in) [] [logger] [https:xxxx]
Mar 1 08:29:33 xxxxxxx info-message(time=2023-03-01T08:29:33.135, appname=abc, user=moon, server=yyy, port=897, msg=logged in) [] [logger] [https:xxxx]", "
")
| mvexpand test
| rename test as _raw
| extract
| fields - _raw _time
| eval _time = strptime(time, "%Y-%m-%dT%H:%M:%S.%3QZ")
| eval forreal = strptime(time, "%Y-%m-%dT%H:%M:%S.%3Q")

The output only has forreal, and no _time

appname	forreal	msg	port	server	time	user
abc	1677684660.137000	logged	123	xxx	2023-03-01T07:31:00.137	john
abc	1677688173.135000	logged	897	yyy	2023-03-01T08:29:33.135	moon

[likithgowda]

Thanks for replying to my post. Time in our _raw logs is ending with Z so shared the updated sample logs (below). Do you think the dateparserverbose warning logs are because of un-wanted data ? as you can see from below example they have improper time or no time available. But all scrap events are sent to nullQueue so not really sure why we still see warn logs for those scrap events in our _interna index.  so any thoughts to avoid or fix this ? Appreciate your help!!

Ex : -Line breaker is breaking first _raw event as below

 Mar 1 07:31:00 xxxxxxx info-message

(time=

2023-03-01T07:31:00.137Z, appname=abc, user=john, server=xxx, port=123, msg=logged in

) [] [logger] [https:xxxx]

Updated sample raw event logs:
Mar 1 07:31:00 xxxxxxx info-message(time=2023-03-01T07:31:00.137Z, appname=abc, user=john, server=xxx, port=123, msg=logged in) [] [logger] [https:xxxx]
Mar 1 08:29:33 xxxxxxx info-message(time=2023-03-01T08:29:33.135Z, appname=abc, user=moon, server=yyy, port=897, msg=logged in) [] [logger] [https:xxxx]

[yuanliu]

You are correct.  Using line breaker that way doesn't make the unwanted data disappear.  The indexer still have to ingest them.  I was wondering what you intend to do with those trims, but didn't think of the error they trigger.

Why would you want to trim your raw data, anyway?  That's generally a bad idea.