Getting Data In

value appearing twice for JSON, none of the old answers work

yohhpark
Path Finder

using UF to send json file and below are the props.conf.

[test_json]
pulldown_type = true
LINE_BREAKER = ([\r\n]+)
INDEXED_EXTRACTIONS = json
KV_MODE = none
SHOULD_LINEMERGE = true
AUTO_KV_JSON = false
category = Structured

and from the inputs.conf also contain

crcSalt = <SOURCE>

 

result keep showing as below

 

AB-17[3]
AB-17[3]
XY-17[2]
XY-17[2]
SI-17[1]
SI-17[1]

 

can't figure out the problem.

Labels (2)
Tags (1)
0 Karma
1 Solution

yohhpark
Path Finder

It was SH that was also extracting.

Putting KV_MODE = none for SH and let the indexer extract should NOT show the duplicate result for Json

View solution in original post

0 Karma

somesoni2
Revered Legend

Did you deploy any props.conf for this sourcetype on your search head? Since you're doing index time field extraction (with INDEXED_EXTRACTIONS = json), there is no need to search time field extraction. I've seen props.conf with search time field extraction (KV_MODE = json) along with index time field extraction causing double extraction. You need to use one. If you just want index time field extraction, explicitly set KV_MODE = none on search head.

0 Karma

yohhpark
Path Finder

result is like such

|table test_id

 

test_id

AB-17[3]
AB-17[3]
XY-17[2]
XY-17[2]
SI-17[1]
SI-17[1]

 

exactly above.

 

200% result. it should 100%, it's extracting twice? i don't know..

 

 

0 Karma

richgalloway
SplunkTrust
SplunkTrust

That is not a query.  A proper query will start with search or some other generating command.

Go to Settings->Fields->Field Extractions to see if you have extractions defined for the sourcetype.

---
If this reply helps you, Karma would be appreciated.
0 Karma

yohhpark
Path Finder

search is not the problem it's the backend, the data is coming in extracted twice. 

thank you. someone will know.

0 Karma

sainag_splunk
Splunk Employee
Splunk Employee

remove the INDEXED_EXTRACTIONS = json on your props. 

0 Karma

yohhpark
Path Finder

@sainag_splunk wrote:

remove the INDEXED_EXTRACTIONS = json on your props. 


I've tried and actually ended up not extracting at all. 

0 Karma

sainag_splunk
Splunk Employee
Splunk Employee

please paste your btool output for props on the UF & enterprise, here to validate. 

splunk btool props list --debug

 

0 Karma

yohhpark
Path Finder

I believe it's because the data is being extracted at index and search time?

 

is there a way for me to stop one or the other?

 

😞 i believe you're on the right track

Tags (1)
0 Karma

yohhpark
Path Finder

/opt/splunkforwarder/etc/system/default/props.conf [_json]
/opt/splunkforwarder/etc/system/default/props.conf INDEXED_EXTRACTIONS = json
/opt/splunkforwarder/etc/system/default/props.conf description = JavaScript Object Notation format. For more information, visit http://json.org/
/opt/splunkforwarder/etc/apps/armor/local/props.conf [armor_json_02]
/opt/splunkforwarder/etc/apps/armor/local/props.conf INDEXED_EXTRACTIONS = json
/opt/splunkforwarder/etc/apps/armor/local/props.conf description = JavaScript Object Notation format. For more information, visit http://json.org/
/opt/splunkforwarder/etc/system/default/props.conf INDEXED_EXTRACTIONS = json
/opt/splunkforwarder/etc/system/default/props.conf [json_no_timestamp]
/opt/splunkforwarder/etc/system/default/props.conf INDEXED_EXTRACTIONS = json
/opt/splunkforwarder/etc/apps/splunk_internal_metrics/default/props.conf [log2metrics_json]
/opt/splunkforwarder/etc/apps/splunk_internal_metrics/default/props.conf INDEXED_EXTRACTIONS = json
/opt/splunkforwarder/etc/apps/splunk_internal_metrics/default/props.conf METRIC-SCHEMA-TRANSFORMS = metric-schema:log2metrics_default_json
/opt/splunkforwarder/etc/apps/splunk_internal_metrics/default/props.conf description = JSON-formatted data. Log-to-metrics processing converts the numeric values in json keys into metric data points.
/opt/splunkforwarder/etc/system/default/props.conf INDEXED_EXTRACTIONS = json
/opt/splunkforwarder/etc/system/default/props.conf INDEXED_EXTRACTIONS = json
/opt/splunkforwarder/etc/system/default/props.conf INDEXED_EXTRACTIONS = json
/opt/splunkforwarder/etc/system/default/props.conf INDEXED_EXTRACTIONS = json
/opt/splunkforwarder/etc/system/default/props.conf INDEXED_EXTRACTIONS = json
/opt/splunkforwarder/etc/system/default/props.conf INDEXED_EXTRACTIONS = json
/opt/splunkforwarder/etc/system/default/props.conf INDEXED_EXTRACTIONS = json

 

 

 

json only extraction

 

all props.txt is way too long.

0 Karma

sainag_splunk
Splunk Employee
Splunk Employee

I don't see anything local other than the below: Not sure if this is your sourcetype.


/opt/splunkforwarder/etc/apps/armor/local/props.conf [armor_json_02]
/opt/splunkforwarder/etc/apps/armor/local/props.conf INDEXED_EXTRACTIONS = json
/opt/splunkforwarder/etc/apps/armor/local/props.conf description = JavaScript Object Notation format. For more information, visit http://json.org/

We need more on what's applying to the enterprise as well. And It's hard to convey the troubleshooting steps here. 

1.) Try to run the btool command specific to your sourcetype such as 

splunk btool props list "your_sourcetype" --debug

     splunk btool props list --debug | grep -v /system/default 

2.) As @somesoni2 mentioned make sure KV_MODE=JSON or INDEXED_EXTRACTIONS = json  only one of them is set . My personal recommendation is to use KV_MODE=JSON instead of I_E=JSON
I hope running this search might help you with the settings applied to the parsing instance.

| rest splunk_server=local /services/configs/conf-props/YOUR_SOURCETYPE
| transpose | search column=eai:acl.app


 Hope this helps. If you need more assistance, encourage you to open an ODS request. an https://www.splunk.com/en_us/pdfs/professional-services/splunk-ondemand-services-portal.pdf

yohhpark
Path Finder

1) 
/opt/splunkforwarder/etc/apps/armor/local/props.conf [armor_json_02]
/opt/splunkforwarder/etc/apps/armor/local/props.conf AUTO_KV_JSON = false
/opt/splunkforwarder/etc/apps/armor/local/props.conf CHARSET = UTF-8
/opt/splunkforwarder/etc/apps/armor/local/props.conf INDEXED_EXTRACTIONS = json
/opt/splunkforwarder/etc/apps/armor/local/props.conf KV_MODE = none
/opt/splunkforwarder/etc/apps/armor/local/props.conf LINE_BREAKER = ([\r\n]+)
/opt/splunkforwarder/etc/apps/armor/local/props.conf NO_BINARY_CHECK = true
/opt/splunkforwarder/etc/apps/armor/local/props.conf SHOULD_LINEMERGE = true
/opt/splunkforwarder/etc/apps/armor/local/props.conf category = Structured
/opt/splunkforwarder/etc/apps/armor/local/props.conf description = JavaScript Object Notation format. For more information, visit http://json.org/
/opt/splunkforwarder/etc/apps/armor/local/props.conf disabled = false
/opt/splunkforwarder/etc/apps/armor/local/props.conf pulldown_type = true

 

2)

i'll try using KV_MODE for JSON isnetad of I_E now.

 

 

0 Karma

sainag_splunk
Splunk Employee
Splunk Employee

You can go to the UI > Settings > Sourcetypes > armor_json_02 > update the KV_MODE=JSON after disabling the I_E

yohhpark
Path Finder

I have that sourcetype setup on the forwarder side.

 

on indexer/SH, can't find that specific sourcetype. Should I had to have the props.conf on the indexer too?

 

IF you mean to update the props.conf to show as KV_MODE = JSON and disable the I_E, iv'e done it on the fowarder side already.


UPDATE

jusdt found this

* When 'INDEXED_EXTRACTIONS = JSON' for a particular source type, do not also 
  set 'KV_MODE = json' for that source type. This causes the Splunk software to 
  extract the JSON fields twice: once at index time, and again at search time.


should I still not use IE?

0 Karma

sainag_splunk
Splunk Employee
Splunk Employee

 KV_MODE=JSON is the search time setting it should be on the SH, you can create a new one from the UI to test. 

yohhpark
Path Finder

before I go ahead, correct me if i don't understand correctly.

 

From the forwarder,

props.conf > remove I_E and add KV_MODE = json

THEN

from the indexer,

create same props.conf from above and keep KV_MODE = json


OR

delete one from forwarder and keep one from the SH (indexer)?

0 Karma

sainag_splunk
Splunk Employee
Splunk Employee

just set the KV_MODE=JSON on the SH (indexer) and remove the I_E from the forwarder.

 
 
0 Karma

yohhpark
Path Finder

any update? I've replied before.

0 Karma

yohhpark
Path Finder

root@armor-index:/opt/splunk/etc/system/local# cat props.conf
[armor_json_02]
KV_MODE = json

 

 

root@armor-uf:/opt/splunkforwarder/etc/apps/armor/local# cat props.conf
[armor_json_02]
SHOULD_LINEMERGE = true
LINE_BREAKER = ([\r\n]+)
NO_BINARY_CHECK = true
CHARSET = UTF-8
#INDEXED_EXTRACTIONS = json
KV_MODE = json
category = Structured
description = JavaScript Object Notation format. For more information, visit http://json.org/
disabled = false
pulldown_type = true
AUTO_KV_JSON = false

 

 

set let me test

getting same results

 

0 Karma

feichinger
Path Finder

I know this post is super old but just for the sake of having another possible solution written down somewhere, the following has solved it for me (based on what was discussed in this thread):

keep the sourcetype in the universal forwarder's app props.conf with INDEXED_EXTRACTIONS = json

[HurricaneMTA_Advanced]
DATETIME_CONFIG =
INDEXED_EXTRACTIONS = json
LINE_BREAKER = ([\r\n]+)
NO_BINARY_CHECK = true
category = Structured
pulldown_type = true
TIMESTAMP_FIELDS = Timestamp
TIME_FORMAT = %FT%T.%7N%:z
SHOULD_LINEMERGE = true
KV_MODE = none
disabled = false
 
Add a sourcetype in the props.conf in some app on the search head with KV_MODE set to none:
 
[HurricaneMTA_Advanced]
KV_MODE = none
disabled = false

yohhpark
Path Finder

It was SH that was also extracting.

Putting KV_MODE = none for SH and let the indexer extract should NOT show the duplicate result for Json

0 Karma
Get Updates on the Splunk Community!

Introduction to Splunk Observability Cloud - Building a Resilient Hybrid Cloud

Introduction to Splunk Observability Cloud - Building a Resilient Hybrid Cloud  In today’s fast-paced digital ...

Observability protocols to know about

Observability protocols define the specifications or formats for collecting, encoding, transporting, and ...

Take Your Breath Away with Splunk Risk-Based Alerting (RBA)

WATCH NOW!The Splunk Guide to Risk-Based Alerting is here to empower your SOC like never before. Join Haylee ...