Why did Splunk fail to parse the timestamp in first MAX_TIMESTAMP_LOOKAHEAD?

[iamsplunker]

Hello, 

I have onboarded the data into Splunk which we have multiple timestamps in the event in different formats. I believe my props settings are correct however it's giving an error in Splunkd.log. Please Advise

Error Details :

DateParserVerbose [99999 merging_0] - Failed to parse timestamp in first MAX_TIMESTAMP_LOOKAHEAD (16) characters of event. Defaulting to timestamp of previous event

Event Details: 

Jul 10 14:19:08 abcdefgh81 dnsmask Jul 10 14:19:08 dnsmask[1520]: cached abcdefg43.wellness.com is 10.220.200.72

Jul 10 14:19:08 abcdefgh81 dnsmask -- [10/July/2022:18:10:10 -9900] dnsmask[1520]: cached abcdefg43.wellness.com is 10.220.200.72

Here are my props settings

TIME_PREFIX=^

TIME_FORMAT=%b %d %H:%M:%S

MAX_TIMESTAMP_LOOKAHEAD = 16

[richgalloway]

Are those *all* of the props for that sourcetype?  Are they in the *right* sourcetype for that data?  Have you used btool (splunk btool --debug props list <<sourcetype>>) to ensure the settings aren't being overridden by another app?

On which instance are the props defined?  They should be on all indexers and heavy forwarders (if any).  Did you restart the instances after loading the props?

[iamsplunker]

@richgalloway  Yes , The sourcetype is created exclusively for this data/app. The interesting part is yes we have multiple time formats in same source coming from lot of servers.

My understanding is If we have multiple time formats in the event it should look at beginning of the event as I mentioned in TIME_PREFIX 

[richgalloway]

Thank you for the response, but it does not provide the information I need to answer the question.

What settings are in props.conf for the sourcetype other than the 3 mentioned? 

Does the sourcetype name in props.conf match the sourcetype name in inputs.conf?

Do other props.conf files have the same sourcetype in them?  Use the splunk btool command to see the exact settings Splunk is using for the sourcetype.

Is the props.conf file installed on all indexers and heavy forwarders?  Were those indexers and HFs restarted after receiving the props.conf file?

[iamsplunker]

@richgalloway  Thank you for your response .Other settings are

CHARSET = UTF-8

LINE_BREAKER = ([\r\n]+)

NO_BINARY_CHECK = True

SHOULD_LINEMERGE = False

TZ = UTC

-- Yes props.conf sourcetype match with inputs.conf sourcetype

There is only 1 sourcetype created for this . This is Splunk Cloud Env

[richgalloway]

Thanks for the info.  Those props look fine.  Are they installed on indexers as well as HFs?