topic Re: Why are my json fields extracted twice? in Splunk Search

Why are my json fields extracted twice?

yannK — Wed, 22 Oct 2014 22:35:37 GMT

I have json events like : { A:"1",B:"2",C:"3"}
with a sourcetype named json_app

When I search the fields, I get 2 values , but not the source/sourcetype/index/_time
A =" 1 , 1" like multivalues

This started in splunk 6.1

Re: Why are my json fields extracted twice?

yannK — Wed, 22 Oct 2014 22:39:41 GMT

Found the explanation, this is the new feature "INDEXED_EXTRACTIONS"
http://docs.splunk.com/Documentation/Splunk/6.1.4/Data/Extractfieldsfromfileheadersatindextime

This does the index time extraction of the fields, at parsing time, this means that the FORWARDERS are parsing the events if specified.
Remark : It also means that for those special events, the timestamp is now extracted by the forwarder, and that the filtering is done by the forwarders

example props.conf on the forwarder

[json_app]
INDEXED_EXTRACTIONS=json

My problem is that I also have a search-time automatic extraction of the fields for json data.

[json_app]
KV_MODE=json

So at the end I got 2 times the same fields.

To fix the issue, I simply disabled the KV_MODE on the search-head, and reloaded with the search command | extract reload=true

Here is my final props for my sourcetype.

[json_app]
INDEXED_EXTRACTIONS=json
KV_MODE=none

Re: Why are my json fields extracted twice?

laserval — Tue, 02 Dec 2014 11:04:09 GMT

Remark : It also means that for those special events, the timestamp is now extracted by the forwarder, and that the filtering is done by the forwarders

This sounds confusing - do you mean Light/Heavy forwarders or Universal Forwarders? The wiki (http://wiki.splunk.com/Where_do_I_configure_my_Splunk_settings%3F) does mention the same thing, that INDEXED_EXTRACTIONS is done in the input stage. This seems an unfortunate effect, have you seen any increased workload on the UF?

Re: Why are my json fields extracted twice?

PatG_ — Fri, 23 Sep 2016 15:18:14 GMT

Thanks a lot!
I was getting crazy trying to understand why that was happening.

Re: Why are my json fields extracted twice?

vinit_masaun — Tue, 29 Sep 2020 17:48:20 GMT

I have the same problem where I am getting duplicated field values from my json logs. I have a universal forwarder that sends data to a heavy forwarder, which then sends that data to indexers. I have the following props.conf in each layer where the INDEXED_EXTRACTIONS is set to 'json' at the universal forwarder and set to none at every other layer (heavy forwarder, indexer, and search head). I don't understand why the fields are still getting extracted twice:

universal forwarder props.conf:
INDEXED_EXTRACTIONS = json
KV_MODE = none
CHARSET = UTF-8
SHOULD_LINEMERGE = true
NO_BINARY_CHECK = true
TRUNCATE = 500000
pulldown_type = true
category = Structured
description = CAP - Ramp Document Monitoring
AUTO_KV_JSON = false

Heavy forwarder props.conf:
INDEXED_EXTRACTIONS = none
KV_MODE = none
CHARSET = UTF-8
SHOULD_LINEMERGE = true
NO_BINARY_CHECK = true
TRUNCATE = 500000
pulldown_type = true
category = Structured
description = CAP - Ramp Document Monitoring
AUTO_KV_JSON = false

Indexer props.conf:
INDEXED_EXTRACTIONS = none
KV_MODE = none
CHARSET = UTF-8
SHOULD_LINEMERGE = true
NO_BINARY_CHECK = true
TRUNCATE = 500000
pulldown_type = true
category = Structured
description = CAP - Ramp Document Monitoring
AUTO_KV_JSON = false

Search Head props.conf:
INDEXED_EXTRACTIONS = none
KV_MODE = none
CHARSET = UTF-8
SHOULD_LINEMERGE = true
NO_BINARY_CHECK = true
TRUNCATE = 500000
pulldown_type = true
category = Structured
description = CAP - Ramp Document Monitoring
AUTO_KV_JSON = false

Re: Why are my json fields extracted twice?

Anam — Thu, 25 Jan 2018 19:00:39 GMT

Hi Vinit

It would be better if you post this as a new question, since this post is 3 years old and you might not get as much visibility on your question.

Thanks