Getting Data In

Events from same file getting separate timestamps

wwhite12
Path Finder

I have json files that have multiple events per file. However when I ingest the data, Splunk parses some of the timestamps correctly and gives other events from the same file the timestamp of when the data was indexed. Anyone else had this problem and know a solution/explanation? All-time search of the source(which is path name that ends with json filename) in picture to show resultsalt text
Thanks in advance
props.conf
[sourcetype]
SHOULD_LINEMERGE=true
LINE_BREAKER=([\r\n]+)
NO_BINARY_CHECK=true
AUTO_KV_JSON=false
CHARSET=UTF-8
INDEXED_EXTRACTIONS=json
KV_MODE=none
TRUNCATE=20000
category=Structured
description=JavaScript Object Notation format. For more information, visit http://json.org/
disabled=false
pulldown_type=true
TIME_PREFIX="date":+

0 Karma
1 Solution

richgalloway
SplunkTrust
SplunkTrust

Try these props. Since the timestamps in your data contain milliseconds, the TIME_FORMAT setting may not work and may need to be dropped.

[sourcetype]
SHOULD_LINEMERGE=false
LINE_BREAKER=([\r\n]+)
NO_BINARY_CHECK=true
AUTO_KV_JSON=false
CHARSET=UTF-8
INDEXED_EXTRACTIONS=json
KV_MODE=none
TRUNCATE=20000
disabled=false
TIME_PREFIX="date":
TIME_FORMAT = %s
TZ = <time zone of source>
---
If this reply helps you, Karma would be appreciated.

View solution in original post

0 Karma

richgalloway
SplunkTrust
SplunkTrust

Try these props. Since the timestamps in your data contain milliseconds, the TIME_FORMAT setting may not work and may need to be dropped.

[sourcetype]
SHOULD_LINEMERGE=false
LINE_BREAKER=([\r\n]+)
NO_BINARY_CHECK=true
AUTO_KV_JSON=false
CHARSET=UTF-8
INDEXED_EXTRACTIONS=json
KV_MODE=none
TRUNCATE=20000
disabled=false
TIME_PREFIX="date":
TIME_FORMAT = %s
TZ = <time zone of source>
---
If this reply helps you, Karma would be appreciated.
0 Karma

wwhite12
Path Finder

Added these two configs to props and it seems to parse data correctly, I have identified the underlying issue as to why some events were incorrectly parsed though and it has to do with similar data fields, will have to ask that in a separate question though
TIME_FORMAT=%s%3N
MAX_TIMESTAMP_LOOKAHEAD=13

0 Karma

richgalloway
SplunkTrust
SplunkTrust

Your props.conf is missing TIME_FORMAT. Share a sample event and we can tell you what else should be changed.

---
If this reply helps you, Karma would be appreciated.
0 Karma

wwhite12
Path Finder

Here's a generic sample of a file that would have two events. In this example, Splunk would've read the epoch "date" field correctly for one but given the other one an indexed time timestamp.

{"message”:”Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Platea dictumst vestibulum rhoncus est. Mattis rhoncus urna neque viverra justo. Justo eget magna fermentum iaculis eu non diam phasellus. Morbi tempus iaculis urna id volutpat lacus laoreet.“,”type”:”bType”,”level":"INFO","details":{},"date":1585769642062}

{"message”:”blah blah blah”,”type”:”aType”,”level":"INFO","details":{},"date":1585769641953}

0 Karma

to4kawa
Ultra Champion
SHOULD_LINEMERGE=true
LINE_BREAKER=([\r\n]+)

you should modify this.

0 Karma
Get Updates on the Splunk Community!

Enterprise Security Content Update (ESCU) | New Releases

In December, the Splunk Threat Research Team had 1 release of new security content via the Enterprise Security ...

Why am I not seeing the finding in Splunk Enterprise Security Analyst Queue?

(This is the first of a series of 2 blogs). Splunk Enterprise Security is a fantastic tool that offers robust ...

Index This | What are the 12 Days of Splunk-mas?

December 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...