topic Re: DateParserVerbose - Failed to parse timestamp. Defaulting to timestamp of previous event Warning is coming in a JSON data load for EPOCH TIMESTAMP in Getting Data In

DateParserVerbose - Failed to parse timestamp. Defaulting to timestamp of previous event Warning is coming in a JSON data load for EPOCH TIMESTAMP

pbankar — Wed, 28 Feb 2018 10:03:53 GMT

Hello, I'm trying to get the timestamp from the below JSON data. I have two type of JSON data to be indexed in Splunk as below.
NOTE Below JSON is copied from the Splunk event by clicking the - "Show as raw text"

JSON DATA for

sourcetype = [lastFound]

{"vulnerability": null, "lastFound": "1511257640614", "os": null, "authType": ["UNIX_AUTH"], "supportedBy": ["VM", "CA-Linux Agent", "CA-Mac Agent"], "discoveryType": ["AUTHENTICATED"], "port": null, "firstFound": "1511257640614"}

JSON DATA for

sourcetype = [lastScanned]

{"size": 520227288, "created": "1432662156000", "vulnerabilities": {"severity2Count": 2, "severity5Count": 11, "severity3Count": 38, "severity1Count": 0, "severity4Count": 30}, "lastScanned": "1516151267981"}

My prop.conf is

[lastFound]
SHOULD_LINEMERGE=false
KV_MODE=json
INDEXED_EXTRACTIONS= JSON
DATETIME_CONFIG = CURRENT
MAX_EVENTS = 50000
TIME_PREFIX=lastFound:
TIME_FORMAT=%s%3N
MAX_TIMESTAMP_LOOKAHEAD=13
TZ=UTC

[lastScanned]
SHOULD_LINEMERGE=false
KV_MODE=json
INDEXED_EXTRACTIONS= JSON
DATETIME_CONFIG = CURRENT
MAX_EVENTS = 50000
TIME_PREFIX=lastScanned:
TIME_FORMAT=%s%3N
MAX_TIMESTAMP_LOOKAHEAD=13
TZ=UTC

I'm curious sometime it works to give me the eventdate as per the "lastFound" but sometime it dosen't. Is it related to the WARNING I'm getting?

P.S. "lastScanned" never worked!!! Please do suggest me if I'm doing anything unexpected.

Re: DateParserVerbose - Failed to parse timestamp. Defaulting to timestamp of previous event Warning is coming in a JSON data load for EPOCH TIMESTAMP

p_gurav — Tue, 29 Sep 2020 18:17:01 GMT

hi ,

Try for last found:
TIME_PREFIX = \"lastFound\":\s
TIME_FORMAT = %s

Try for last scanned:
TIME_PREFIX = \"lastScanned\":\s
TIME_FORMAT = %s

Re: DateParserVerbose - Failed to parse timestamp. Defaulting to timestamp of previous event Warning is coming in a JSON data load for EPOCH TIMESTAMP

pbankar — Mon, 05 Mar 2018 05:43:53 GMT

Hi @p_gurav,

Thanks for your comment. I tried below as your sugesstion was good but didn't worked for me.

[lastFound]
 SHOULD_LINEMERGE=false
 KV_MODE=json
 INDEXED_EXTRACTIONS= JSON
 DATETIME_CONFIG = CURRENT
 MAX_EVENTS = 50000
 TIME_PREFIX=\"lastFound\":\s
 TIME_FORMAT=%s%3N
 #MAX_TIMESTAMP_LOOKAHEAD=13 # commented this line. So it worked for the lastFound
 TZ=UTC

 [lastScanned]
 SHOULD_LINEMERGE=false
 KV_MODE=json
 INDEXED_EXTRACTIONS= JSON
 DATETIME_CONFIG = CURRENT
 MAX_EVENTS = 50000
 TIME_PREFIX=\"lastScanned\":\s
 TIME_FORMAT=%s%3N
# MAX_TIMESTAMP_LOOKAHEAD=13 # commented same but not working here. Not sure Why?
 TZ=UTC

Re: DateParserVerbose - Failed to parse timestamp. Defaulting to timestamp of previous event Warning is coming in a JSON data load for EPOCH TIMESTAMP

p_gurav — Tue, 29 Sep 2020 18:18:07 GMT

Hi,

Can you try :

[sample_json]
DATETIME_CONFIG =
INDEXED_EXTRACTIONS = json
KV_MODE = none
NO_BINARY_CHECK = true
TIMESTAMP_FIELDS = lastFound
TIME_FORMAT = %s%3N
category = Structured
description = JavaScript Object Notation format. For more information, visit http://json.org/
disabled = false
pulldown_type = true

[sample_json1]
DATETIME_CONFIG =
INDEXED_EXTRACTIONS = json
KV_MODE = none
NO_BINARY_CHECK = true
TIMESTAMP_FIELDS = lastScanned
TIME_FORMAT = %s%3N
category = Structured
description = JavaScript Object Notation format. For more information, visit http://json.org/
disabled = false
pulldown_type = true

Re: DateParserVerbose - Failed to parse timestamp. Defaulting to timestamp of previous event Warning is coming in a JSON data load for EPOCH TIMESTAMP

pbankar — Mon, 05 Mar 2018 11:51:29 GMT

@p_gurav

Below is working for me:

[lastScanned]
INDEXED_EXTRACTIONS=json
KV_MODE=json
MAX_EVENTS=50000
NO_BINARY_CHECK=true
SHOULD_LINEMERGE=true
TIMESTAMP_FIELDS=lastScanned
TIME_FORMAT=%s%3N
category=Structured
description=JavaScript Object Notation format. For more information, visit http://json.org/
disabled=false
pulldown_type=true

Re: DateParserVerbose - Failed to parse timestamp. Defaulting to timestamp of previous event Warning is coming in a JSON data load for EPOCH TIMESTAMP

p_gurav — Mon, 05 Mar 2018 11:56:12 GMT

Great. Happy to help!!!

Re: DateParserVerbose - Failed to parse timestamp. Defaulting to timestamp of previous event Warning is coming in a JSON data load for EPOCH TIMESTAMP

pbankar — Mon, 05 Mar 2018 11:58:31 GMT

Thanks @p_gurav

Re: DateParserVerbose - Failed to parse timestamp. Defaulting to timestamp of previous event Warning is coming in a JSON data load for EPOCH TIMESTAMP

pbankar — Tue, 29 Sep 2020 18:21:00 GMT

@p_gurav I need one more help from you.

In both of the JSON, there are 2 epoch time fields.

sourcetype = [lastFound]

{"vulnerability": null, "lastFound": "1511257640614", "os": null, "authType": ["UNIX_AUTH"], "supportedBy": ["VM", "CA-Linux Agent", "CA-Mac Agent"], "discoveryType": ["AUTHENTICATED"], "port": null, "firstFound": "1511257640614"}

JSON DATA for

sourcetype = [lastScanned]

{"size": 520227288, "created": "1432662156000", "vulnerabilities": {"severity2Count": 2, "severity5Count": 11, "severity3Count": 38, "severity1Count": 0, "severity4Count": 30}, "lastScanned": "1516151267981"}

I need to provide an OR case for the TIMESTAMP_FIELDS.
I tried this but it didn't work.
TIMESTAMP_FIELDS=lastScanned|created

Could you suggest something?