Getting Data In

DateParserVerbose - Failed to parse timestamp. Defaulting to timestamp of previous event Warning is coming in a JSON data load for EPOCH TIMESTAMP

pbankar
Path Finder

Hello, I'm trying to get the timestamp from the below JSON data. I have two type of JSON data to be indexed in Splunk as below.
NOTE Below JSON is copied from the Splunk event by clicking the - "Show as raw text"

JSON DATA for

sourcetype = [lastFound]

{"vulnerability": null, "lastFound": "1511257640614", "os": null, "authType": ["UNIX_AUTH"], "supportedBy": ["VM", "CA-Linux Agent", "CA-Mac Agent"], "discoveryType": ["AUTHENTICATED"], "port": null, "firstFound": "1511257640614"}

JSON DATA for

sourcetype = [lastScanned]

{"size": 520227288, "created": "1432662156000", "vulnerabilities": {"severity2Count": 2, "severity5Count": 11, "severity3Count": 38, "severity1Count": 0, "severity4Count": 30}, "lastScanned": "1516151267981"}

My prop.conf is

[lastFound]
SHOULD_LINEMERGE=false
KV_MODE=json
INDEXED_EXTRACTIONS= JSON
DATETIME_CONFIG = CURRENT
MAX_EVENTS = 50000
TIME_PREFIX=lastFound:
TIME_FORMAT=%s%3N
MAX_TIMESTAMP_LOOKAHEAD=13
TZ=UTC

[lastScanned]
SHOULD_LINEMERGE=false
KV_MODE=json
INDEXED_EXTRACTIONS= JSON
DATETIME_CONFIG = CURRENT
MAX_EVENTS = 50000
TIME_PREFIX=lastScanned:
TIME_FORMAT=%s%3N
MAX_TIMESTAMP_LOOKAHEAD=13
TZ=UTC

I'm curious sometime it works to give me the eventdate as per the "lastFound" but sometime it dosen't. Is it related to the WARNING I'm getting?

P.S. "lastScanned" never worked!!! Please do suggest me if I'm doing anything unexpected.

Tags (1)
0 Karma
1 Solution

p_gurav
Champion

hi ,

Try for last found:
TIME_PREFIX = \"lastFound\":\s
TIME_FORMAT = %s

Try for last scanned:
TIME_PREFIX = \"lastScanned\":\s
TIME_FORMAT = %s

View solution in original post

0 Karma

p_gurav
Champion

hi ,

Try for last found:
TIME_PREFIX = \"lastFound\":\s
TIME_FORMAT = %s

Try for last scanned:
TIME_PREFIX = \"lastScanned\":\s
TIME_FORMAT = %s

View solution in original post

0 Karma

pbankar
Path Finder

Hi @p_gurav,

Thanks for your comment. I tried below as your sugesstion was good but didn't worked for me.

[lastFound]
 SHOULD_LINEMERGE=false
 KV_MODE=json
 INDEXED_EXTRACTIONS= JSON
 DATETIME_CONFIG = CURRENT
 MAX_EVENTS = 50000
 TIME_PREFIX=\"lastFound\":\s
 TIME_FORMAT=%s%3N
 #MAX_TIMESTAMP_LOOKAHEAD=13 # commented this line. So it worked for the lastFound
 TZ=UTC

 [lastScanned]
 SHOULD_LINEMERGE=false
 KV_MODE=json
 INDEXED_EXTRACTIONS= JSON
 DATETIME_CONFIG = CURRENT
 MAX_EVENTS = 50000
 TIME_PREFIX=\"lastScanned\":\s
 TIME_FORMAT=%s%3N
# MAX_TIMESTAMP_LOOKAHEAD=13 # commented same but not working here. Not sure Why?
 TZ=UTC
0 Karma

p_gurav
Champion

Hi,

Can you try :

[sample_json]
DATETIME_CONFIG =
INDEXED_EXTRACTIONS = json
KV_MODE = none
NO_BINARY_CHECK = true
TIMESTAMP_FIELDS = lastFound
TIME_FORMAT = %s%3N
category = Structured
description = JavaScript Object Notation format. For more information, visit http://json.org/
disabled = false
pulldown_type = true

[sample_json1]
DATETIME_CONFIG =
INDEXED_EXTRACTIONS = json
KV_MODE = none
NO_BINARY_CHECK = true
TIMESTAMP_FIELDS = lastScanned
TIME_FORMAT = %s%3N
category = Structured
description = JavaScript Object Notation format. For more information, visit http://json.org/
disabled = false
pulldown_type = true

0 Karma

pbankar
Path Finder

@p_gurav I need one more help from you.

In both of the JSON, there are 2 epoch time fields.

sourcetype = [lastFound]

{"vulnerability": null, "lastFound": "1511257640614", "os": null, "authType": ["UNIX_AUTH"], "supportedBy": ["VM", "CA-Linux Agent", "CA-Mac Agent"], "discoveryType": ["AUTHENTICATED"], "port": null, "firstFound": "1511257640614"}

JSON DATA for

sourcetype = [lastScanned]

{"size": 520227288, "created": "1432662156000", "vulnerabilities": {"severity2Count": 2, "severity5Count": 11, "severity3Count": 38, "severity1Count": 0, "severity4Count": 30}, "lastScanned": "1516151267981"}

I need to provide an OR case for the TIMESTAMP_FIELDS.
I tried this but it didn't work.
TIMESTAMP_FIELDS=lastScanned|created

Could you suggest something?

0 Karma

pbankar
Path Finder

@p_gurav

Below is working for me:

[lastScanned]
INDEXED_EXTRACTIONS=json
KV_MODE=json
MAX_EVENTS=50000
NO_BINARY_CHECK=true
SHOULD_LINEMERGE=true
TIMESTAMP_FIELDS=lastScanned
TIME_FORMAT=%s%3N
category=Structured
description=JavaScript Object Notation format. For more information, visit http://json.org/
disabled=false
pulldown_type=true
0 Karma

p_gurav
Champion

Great. Happy to help!!!

0 Karma

pbankar
Path Finder

Thanks @p_gurav

0 Karma
Did you miss .conf21 Virtual?

Good news! The event's keynotes and many of its breakout sessions are now available online, and still totally FREE!