Timestamps are not properly being parsed using v1.0.0 of the TA-pps_ondemand app.
01-16-2019 02:56:09.819 -0500 WARN DateParserVerbose - Failed to parse timestamp in first MAX_TIMESTAMP_LOOKAHEAD (128) characters of event. Defaulting to timestamp of previous event (Wed Jan 16 02:28:43 2019). Context: source=proofpoint_message_log|host=sp-01|pps_messagelog|
Add the following to local/props.conf
[pps_messagelog]
LINE_BREAKER = ([\r\n]+\s*)\{
TIME_PREFIX = \"ts\"\s*:\s*\"
MAX_TIMESTAMP_LOOKAHEAD = 34
TIME_FORMAT = %Y-%m-%dT%H:%M:%S.%6N%z
[pps_maillog]
LINE_BREAKER = ([\r\n]+\s*)\{
TIME_PREFIX = \"ts\"\s*:\s*\"
MAX_TIMESTAMP_LOOKAHEAD = 34
TIME_FORMAT = %Y-%m-%dT%H:%M:%S.%6N%z
added 1 May 2019
Add the following to local/limits.conf
[spath]
extract_all=true
extraction_cutoff=30000
note: version 1.0.2 of the app added "DATETIME_CONFIG = CURRENT" to default/props.conf, which overrides these settings. I edited default to remove that configuration, but "proper" method would be to add "DATETIME_CONFIG = /etc/datetime.xml" to local/props.conf to restore the default. I have reported this to the developers and they responded they're workig on a revised version to include proper date/time extraction.
Add the following to local/props.conf
[pps_messagelog]
LINE_BREAKER = ([\r\n]+\s*)\{
TIME_PREFIX = \"ts\"\s*:\s*\"
MAX_TIMESTAMP_LOOKAHEAD = 34
TIME_FORMAT = %Y-%m-%dT%H:%M:%S.%6N%z
[pps_maillog]
LINE_BREAKER = ([\r\n]+\s*)\{
TIME_PREFIX = \"ts\"\s*:\s*\"
MAX_TIMESTAMP_LOOKAHEAD = 34
TIME_FORMAT = %Y-%m-%dT%H:%M:%S.%6N%z
added 1 May 2019
Add the following to local/limits.conf
[spath]
extract_all=true
extraction_cutoff=30000
note: version 1.0.2 of the app added "DATETIME_CONFIG = CURRENT" to default/props.conf, which overrides these settings. I edited default to remove that configuration, but "proper" method would be to add "DATETIME_CONFIG = /etc/datetime.xml" to local/props.conf to restore the default. I have reported this to the developers and they responded they're workig on a revised version to include proper date/time extraction.
I also had the same issue. I tried setting up timestamp configuration myself but i messed something. Later found that there was slight mistake in the Regex for TIME_PREFIX. I used your suggestion and now I am perfect.
Thanks.
Upvoted.