topic Re: duplicate JSON are coming in search in Getting Data In

duplicate JSON are coming in search

kirrusk — Wed, 02 Dec 2020 16:14:23 GMT

I have a json file like below

{"env":"UAT","label":"jenkins-17887.api.v2.dm.btc","App":"dm-d-services","rlmtemplate":"f2_api_fed","lastupdate":2020-11-23 11:09:78:455,"region":"APAC"}{"env":"UAT","label":"jenkins-17687.api.v2.dm.btc","App":"dt-s-services","rlmtemplate":"f3_api_fed","lastupdate":2020-11-23 11:025:79:475,"region":"APAC"}{"env":"UAT","label":"jenkins-18657.api.v2.dm.btc","App":"dt-s-services","rlmtemplate":"f3_api_fed","lastupdate":2020-11-23 11:025:79:475,"region":"APAC"}{"env":"UAT","label":"jenkins-17637.api.v2.dm.btc","App":"dt-s-services","rlmtemplate":"f3_api_fed","lastupdate":2020-11-23 11:025:79:475,"region":"APAC"}

in splunk,

_raw contains valid json data for all events.

issue is all fields are multi valued containing two copies of the json object.

forwarding json data to splunk and props is installed on indexer not in heavy forwarder,

props.conf

[test_json]

INDEXED_EXTRACTIONS = JSON

KV_MODE = none

AUTO_KV_JSON = false

SHOULD_LINEMERGE = false

FYI , using splunk 8.0

tried by setting KV_MODE = JSON but it is also not working

Re: duplicate JSON are coming in search

richgalloway — Wed, 02 Dec 2020 15:48:43 GMT

In splunk it's showing one event but while applying query on it , like query | table fields
duplicate values are coming in fields

How is Splunk showing one event without a query? What is the query you use that is showing duplicate values?

Re: duplicate JSON are coming in search

kirrusk — Wed, 02 Dec 2020 16:14:54 GMT

@richgalloway updated question

Re: duplicate JSON are coming in search

richgalloway — Wed, 02 Dec 2020 16:47:58 GMT

Are you perhaps using Verbose Mode?

The INDEXED_EXTRACTIONS=json setting tells Splunk to extract fields from each event at index time. A Verbose Mode search then automatically extracts fields at search time. The result is duplicates.

Re: duplicate JSON are coming in search

kirrusk — Thu, 03 Dec 2020 05:20:47 GMT

@richgalloway Can u pls suggest how can i rectify this, it is running in verbose mode. I changed to fast mode and smart mode but duplicates are still coming