topic Re: Events from same file getting separate timestamps in Getting Data In

Events from same file getting separate timestamps

wwhite12 — Wed, 30 Sep 2020 05:26:14 GMT

I have json files that have multiple events per file. However when I ingest the data, Splunk parses some of the timestamps correctly and gives other events from the same file the timestamp of when the data was indexed. Anyone else had this problem and know a solution/explanation? All-time search of the source(which is path name that ends with json filename) in picture to show results
Thanks in advance
props.conf
[sourcetype]
SHOULD_LINEMERGE=true
LINE_BREAKER=([\r\n]+)
NO_BINARY_CHECK=true
AUTO_KV_JSON=false
CHARSET=UTF-8
INDEXED_EXTRACTIONS=json
KV_MODE=none
TRUNCATE=20000
category=Structured
description=JavaScript Object Notation format. For more information, visit http://json.org/
disabled=false
pulldown_type=true
TIME_PREFIX="date":+

Re: Events from same file getting separate timestamps

richgalloway — Thu, 14 May 2020 00:11:42 GMT

Your props.conf is missing TIME_FORMAT. Share a sample event and we can tell you what else should be changed.

Re: Events from same file getting separate timestamps

wwhite12 — Thu, 14 May 2020 00:49:46 GMT

Here's a generic sample of a file that would have two events. In this example, Splunk would've read the epoch "date" field correctly for one but given the other one an indexed time timestamp.

{"message”:”Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Platea dictumst vestibulum rhoncus est. Mattis rhoncus urna neque viverra justo. Justo eget magna fermentum iaculis eu non diam phasellus. Morbi tempus iaculis urna id volutpat lacus laoreet.“,”type”:”bType”,”level":"INFO","details":{},"date":1585769642062}

{"message”:”blah blah blah”,”type”:”aType”,”level":"INFO","details":{},"date":1585769641953}

Re: Events from same file getting separate timestamps

to4kawa — Thu, 14 May 2020 00:54:25 GMT

SHOULD_LINEMERGE=true
LINE_BREAKER=([\r\n]+)

you should modify this.

Re: Events from same file getting separate timestamps

richgalloway — Thu, 14 May 2020 12:29:06 GMT

Try these props. Since the timestamps in your data contain milliseconds, the TIME_FORMAT setting may not work and may need to be dropped.

[sourcetype]
SHOULD_LINEMERGE=false
LINE_BREAKER=([\r\n]+)
NO_BINARY_CHECK=true
AUTO_KV_JSON=false
CHARSET=UTF-8
INDEXED_EXTRACTIONS=json
KV_MODE=none
TRUNCATE=20000
disabled=false
TIME_PREFIX="date":
TIME_FORMAT = %s
TZ = <time zone of source>

Re: Events from same file getting separate timestamps

wwhite12 — Wed, 30 Sep 2020 05:26:33 GMT

Added these two configs to props and it seems to parse data correctly, I have identified the underlying issue as to why some events were incorrectly parsed though and it has to do with similar data fields, will have to ask that in a separate question though
TIME_FORMAT=%s%3N
MAX_TIMESTAMP_LOOKAHEAD=13