topic Re: JSON extra value "none" timestamp field in Getting Data In

JSON extra value "none" timestamp field

kwarre3036 — Tue, 08 Jun 2021 14:06:46 GMT

I am attempting to index and search JSON logs and each event contains an extra value ("none") for timestamp that I would like to eliminate.

Here is my inputs.conf

[monitor:///home/username/json_test.log]
index = index_name
source = json_test.log
sourcetype = json_kwarre_v3
host = myhostname

Here is my props.conf

[json_kwarre_v3]
BREAK_ONLY_BEFORE = ^{
MUST_BREAK_AFTER = }$
LINE_BREAKER = ^{
KV_MODE= json
NO_BINARY_CHECK = true
TIME_PREFIX = timestamp\"\:\ \"
TIME_FORMAT=%Y-%m-%dT%H:%M:%S:%3N
category = Structured
description = A variant of the JSON source type, with support for nonexistent timestamps
disabled = false
pulldown_type = true
BREAK_ONLY_BEFORE_DATE =

Below, I have pasted the json event from the log. The event actually looks like one line in the log, but when pasted into the ticket it appears as several lines.

{"sessionId":"5b8d6d8d-8e63-413b-876e-34cfaa894676","service":"RAF","request":{"vendorId":"Digital","clientId":"2234567890g"},"response":{"vendorId":"Digital","clientId":"2234567890g","transactionStatus":"1000","transactionMessage":"Success"},"routing_time":"10","elapsedTime":"107","timestamp_begin":"2021-06-06T17:51:30.895Z","level":"info","message":"SUCCESS","timestamp":"2021-06-06T17:51:31.002Z"}

I have attached screenshot of my results. The only unexpected result I would like to eliminate is the extra "none" associated with the timestamp field

Re: JSON extra value "none" timestamp field

venkatasri — Wed, 09 Jun 2021 05:04:35 GMT

Hi @kwarre3036

You shall correct the TIME_PREFIX as follows and retest. It's kind of weird could be a splunk bug its not supposed add value none to timestamp event field which is the original source for _time.

TIME_PREFIX = timestamp\"\:\"

----------

An upvote would be appreciated if it helps!

Re: JSON extra value "none" timestamp field

venkatasri — Wed, 09 Jun 2021 05:07:38 GMT

You shall correct your TIME_FORMAT as well.

TIME_FORMAT=%Y-%m-%dT%H:%M:%S.%3N

Re: JSON extra value "none" timestamp field

kwarre3036 — Wed, 09 Jun 2021 13:09:13 GMT

Making these two modifications to the props.conf did eliminate the value "none" in the timestamp field. Now, my timestamp is being parsed and only one value is present. This is good!

However, now my _time = timestamp which is what you indicated in your note. When I had the parameters set the other way, my _time = indextime.

It seems now that I will need to run " | eval indextime=strftime(_indextime,"%Y-%m-%d %H:%M:%S")" in order to see my indextime. This should not be an issue unless for some reason, my events have a large discrepancy between timestamp and indextime.

Re: JSON extra value "none" timestamp field

venkatasri — Wed, 09 Jun 2021 21:02:22 GMT

Hi @kwarre3036

If you can accept the solution for original problem that would be great.

The `_indextime` field contains the time that an event was indexed, expressed in Unix time. You might use this field to focus on or filter out events that were indexed within a specific range of time. The _indextime is a default field available, it's time when Splunk writes the event to disk OR about to write to disk.

_time is timestamp in your event, you have to check monitoring console of splunk to find the reason for delay in indexing or it could be you haven't set the TZ aka timezone correctly. Open a new post for this one.

----

An upvote would be appreciated if it helps!