value appearing twice for JSON, none of the old answers work

[yohhpark]

using UF to send json file and below are the props.conf.

[test_json]
pulldown_type = true
LINE_BREAKER = ([\r\n]+)
INDEXED_EXTRACTIONS = json
KV_MODE = none
SHOULD_LINEMERGE = true
AUTO_KV_JSON = false
category = Structured

and from the inputs.conf also contain

crcSalt = <SOURCE>

result keep showing as below

AB-17[3] AB-17[3]

XY-17[2] XY-17[2]

SI-17[1] SI-17[1]

can't figure out the problem.

[somesoni2]

Did you deploy any props.conf for this sourcetype on your search head? Since you're doing index time field extraction (with INDEXED_EXTRACTIONS = json), there is no need to search time field extraction. I've seen props.conf with search time field extraction (KV_MODE = json) along with index time field extraction causing double extraction. You need to use one. If you just want index time field extraction, explicitly set KV_MODE = none on search head.

[yohhpark]

result is like such

|table test_id

test_id

AB-17[3] AB-17[3]

XY-17[2] XY-17[2]

SI-17[1] SI-17[1]

exactly above.

200% result. it should 100%, it's extracting twice? i don't know..

[richgalloway]

That is not a query.  A proper query will start with search or some other generating command.

Go to Settings->Fields->Field Extractions to see if you have extractions defined for the sourcetype.

[yohhpark]

search is not the problem it's the backend, the data is coming in extracted twice. 

thank you. someone will know.

[sainag_splunk]

remove the INDEXED_EXTRACTIONS = json on your props. 

[yohhpark]

---

@sainag_splunk wrote:

remove the INDEXED_EXTRACTIONS = json on your props. 

---

I've tried and actually ended up not extracting at all. 

[sainag_splunk]

please paste your btool output for props on the UF & enterprise, here to validate. 

splunk btool props list --debug

[yohhpark]

I believe it's because the data is being extracted at index and search time?

is there a way for me to stop one or the other?

😞 i believe you're on the right track

[yohhpark]

/opt/splunkforwarder/etc/system/default/props.conf [_json]
/opt/splunkforwarder/etc/system/default/props.conf INDEXED_EXTRACTIONS = json
/opt/splunkforwarder/etc/system/default/props.conf description = JavaScript Object Notation format. For more information, visit http://json.org/
/opt/splunkforwarder/etc/apps/armor/local/props.conf [armor_json_02]
/opt/splunkforwarder/etc/apps/armor/local/props.conf INDEXED_EXTRACTIONS = json
/opt/splunkforwarder/etc/apps/armor/local/props.conf description = JavaScript Object Notation format. For more information, visit http://json.org/
/opt/splunkforwarder/etc/system/default/props.conf INDEXED_EXTRACTIONS = json
/opt/splunkforwarder/etc/system/default/props.conf [json_no_timestamp]
/opt/splunkforwarder/etc/system/default/props.conf INDEXED_EXTRACTIONS = json
/opt/splunkforwarder/etc/apps/splunk_internal_metrics/default/props.conf [log2metrics_json]
/opt/splunkforwarder/etc/apps/splunk_internal_metrics/default/props.conf INDEXED_EXTRACTIONS = json
/opt/splunkforwarder/etc/apps/splunk_internal_metrics/default/props.conf METRIC-SCHEMA-TRANSFORMS = metric-schema:log2metrics_default_json
/opt/splunkforwarder/etc/apps/splunk_internal_metrics/default/props.conf description = JSON-formatted data. Log-to-metrics processing converts the numeric values in json keys into metric data points.
/opt/splunkforwarder/etc/system/default/props.conf INDEXED_EXTRACTIONS = json
/opt/splunkforwarder/etc/system/default/props.conf INDEXED_EXTRACTIONS = json
/opt/splunkforwarder/etc/system/default/props.conf INDEXED_EXTRACTIONS = json
/opt/splunkforwarder/etc/system/default/props.conf INDEXED_EXTRACTIONS = json
/opt/splunkforwarder/etc/system/default/props.conf INDEXED_EXTRACTIONS = json
/opt/splunkforwarder/etc/system/default/props.conf INDEXED_EXTRACTIONS = json
/opt/splunkforwarder/etc/system/default/props.conf INDEXED_EXTRACTIONS = json

json only extraction

all props.txt is way too long.

[sainag_splunk]

I don't see anything local other than the below: Not sure if this is your sourcetype.

/opt/splunkforwarder/etc/apps/armor/local/props.conf [armor_json_02]
/opt/splunkforwarder/etc/apps/armor/local/props.conf INDEXED_EXTRACTIONS = json
/opt/splunkforwarder/etc/apps/armor/local/props.conf description = JavaScript Object Notation format. For more information, visit http://json.org/

We need more on what's applying to the enterprise as well. And It's hard to convey the troubleshooting steps here. 

1.) Try to run the btool command specific to your sourcetype such as 

splunk btool props list "your_sourcetype" --debug

     splunk btool props list --debug | grep -v /system/default 

2.) As @somesoni2 mentioned make sure KV_MODE=JSON or INDEXED_EXTRACTIONS = json  only one of them is set . My personal recommendation is to use KV_MODE=JSON instead of I_E=JSON
I hope running this search might help you with the settings applied to the parsing instance.

| rest splunk_server=local /services/configs/conf-props/YOUR_SOURCETYPE
| transpose | search column=eai:acl.app

 Hope this helps. If you need more assistance, encourage you to open an ODS request. an https://www.splunk.com/en_us/pdfs/professional-services/splunk-ondemand-services-portal.pdf

[yohhpark]

1) 
/opt/splunkforwarder/etc/apps/armor/local/props.conf [armor_json_02]
/opt/splunkforwarder/etc/apps/armor/local/props.conf AUTO_KV_JSON = false
/opt/splunkforwarder/etc/apps/armor/local/props.conf CHARSET = UTF-8
/opt/splunkforwarder/etc/apps/armor/local/props.conf INDEXED_EXTRACTIONS = json
/opt/splunkforwarder/etc/apps/armor/local/props.conf KV_MODE = none
/opt/splunkforwarder/etc/apps/armor/local/props.conf LINE_BREAKER = ([\r\n]+)
/opt/splunkforwarder/etc/apps/armor/local/props.conf NO_BINARY_CHECK = true
/opt/splunkforwarder/etc/apps/armor/local/props.conf SHOULD_LINEMERGE = true
/opt/splunkforwarder/etc/apps/armor/local/props.conf category = Structured
/opt/splunkforwarder/etc/apps/armor/local/props.conf description = JavaScript Object Notation format. For more information, visit http://json.org/
/opt/splunkforwarder/etc/apps/armor/local/props.conf disabled = false
/opt/splunkforwarder/etc/apps/armor/local/props.conf pulldown_type = true

2)

i'll try using KV_MODE for JSON isnetad of I_E now.

[sainag_splunk]

You can go to the UI > Settings > Sourcetypes > armor_json_02 > update the KV_MODE=JSON after disabling the I_E

[yohhpark]

I have that sourcetype setup on the forwarder side.

on indexer/SH, can't find that specific sourcetype. Should I had to have the props.conf on the indexer too?

IF you mean to update the props.conf to show as KV_MODE = JSON and disable the I_E, iv'e done it on the fowarder side already.


UPDATE

jusdt found this

* When 'INDEXED_EXTRACTIONS = JSON' for a particular source type, do not also 
  set 'KV_MODE = json' for that source type. This causes the Splunk software to 
  extract the JSON fields twice: once at index time, and again at search time.

should I still not use IE?

[sainag_splunk]

 KV_MODE=JSON is the search time setting it should be on the SH, you can create a new one from the UI to test. 

[yohhpark]

before I go ahead, correct me if i don't understand correctly.

From the forwarder,

props.conf > remove I_E and add KV_MODE = json

THEN

from the indexer,

create same props.conf from above and keep KV_MODE = json


OR

delete one from forwarder and keep one from the SH (indexer)?

[sainag_splunk]

just set the KV_MODE=JSON on the SH (indexer) and remove the I_E from the forwarder.

[yohhpark]

any update? I've replied before.

[yohhpark]

root@armor-index:/opt/splunk/etc/system/local# cat props.conf
[armor_json_02]
KV_MODE = json

root@armor-uf:/opt/splunkforwarder/etc/apps/armor/local# cat props.conf
[armor_json_02]
SHOULD_LINEMERGE = true
LINE_BREAKER = ([\r\n]+)
NO_BINARY_CHECK = true
CHARSET = UTF-8
#INDEXED_EXTRACTIONS = json
KV_MODE = json
category = Structured
description = JavaScript Object Notation format. For more information, visit http://json.org/
disabled = false
pulldown_type = true
AUTO_KV_JSON = false

set let me test

getting same results

[feichinger]

I know this post is super old but just for the sake of having another possible solution written down somewhere, the following has solved it for me (based on what was discussed in this thread):

keep the sourcetype in the universal forwarder's app props.conf with INDEXED_EXTRACTIONS = json

[HurricaneMTA_Advanced]

DATETIME_CONFIG =

INDEXED_EXTRACTIONS = json

LINE_BREAKER = ([\r\n]+)

NO_BINARY_CHECK = true

category = Structured

pulldown_type = true

TIMESTAMP_FIELDS = Timestamp

TIME_FORMAT = %FT%T.%7N%:z

SHOULD_LINEMERGE = true

KV_MODE = none

disabled = false

Add a sourcetype in the props.conf in some app on the search head with KV_MODE set to none:

[HurricaneMTA_Advanced]

KV_MODE = none

disabled = false

[yohhpark · Accepted solution]

It was SH that was also extracting.

Putting KV_MODE = none for SH and let the indexer extract should NOT show the duplicate result for Json