Getting Data In
Highlighted

Why are JSON fields extracted and displayed twice?

Path Finder

JSON fields are extracted twice.

On Universal forwarder (7.0.3) the settings props.conf are like this

[my_sourcetype]
SHOULD_LINEMERGE=true
LINE_BREAKER=([\r\n]+)
NO_BINARY_CHECK=true
CHARSET=UTF-8
INDEXED_EXTRACTIONS=json
KV_MODE=none
category=Structured
disabled=false
pulldown_type=true
TIMESTAMP_FIELDS=timestamp

On Search Head(7.2.6), tried all combinations of below in props.conf

[my_sourcetype]
INDEXED_EXTRACTIONS=json
KV_MODE=none
AUTO_KV_JSON = false
0 Karma
Highlighted

Re: Why are JSON fields extracted and displayed twice?

Motivator

What do you mean with "JSON fields are extracted twice."?

Also INDEXED_EXTRACTIONS is use during indexing stage in UFs or IDXs. So unless you are indexing data using you search head, there is no point on this particular atribute being there.

check this. It shows where in the indexing pipeline each atribute is used.
https://wiki.splunk.com/Community:HowIndexingWorks

------------
Hope I was able to help you. If so, an upvote would be appreciated.
0 Karma
Highlighted

Re: Why are JSON fields extracted and displayed twice?

Path Finder

Correct. I tried below as well on SH.

[my_sourcetype]
 KV_MODE=none
 AUTO_KV_JSON = false
0 Karma
Highlighted

Re: Why are JSON fields extracted and displayed twice?

Motivator

try to run a btool to check whatever is also being used with you sourcetype
in CLI splunk btool props list --debug my_sourcetype

------------
Hope I was able to help you. If so, an upvote would be appreciated.
0 Karma
Highlighted

Re: Why are JSON fields extracted and displayed twice?

Path Finder

It shows this

    [root@lgpbdus4101 bin]# ./splunk btool props list --debug my_sourcetype
/data/splunk/etc/apps/my_app/local/props.conf [my_sourcetype]
/data/splunk/etc/system/default/props.conf           ADD_EXTRA_TIME_FIELDS = True
/data/splunk/etc/system/default/props.conf           ANNOTATE_PUNCT = True
/data/splunk/etc/apps/my_app/local/props.conf AUTO_KV_JSON = false
/data/splunk/etc/system/default/props.conf           BREAK_ONLY_BEFORE =
/data/splunk/etc/system/default/props.conf           BREAK_ONLY_BEFORE_DATE = True
/data/splunk/etc/system/default/props.conf           CHARSET = UTF-8
/data/splunk/etc/system/default/props.conf           DATETIME_CONFIG = /etc/datetime.xml
/data/splunk/etc/system/default/props.conf           DEPTH_LIMIT = 1000
/data/splunk/etc/system/default/props.conf           HEADER_MODE =
/data/splunk/etc/apps/my_app/local/props.conf KV_MODE = none
/data/splunk/etc/system/default/props.conf           LEARN_MODEL = true
/data/splunk/etc/system/default/props.conf           LEARN_SOURCETYPE = true
/data/splunk/etc/system/default/props.conf           LINE_BREAKER_LOOKBEHIND = 100
/data/splunk/etc/system/default/props.conf           MATCH_LIMIT = 100000
/data/splunk/etc/system/default/props.conf           MAX_DAYS_AGO = 2000
/data/splunk/etc/system/default/props.conf           MAX_DAYS_HENCE = 2
/data/splunk/etc/system/default/props.conf           MAX_DIFF_SECS_AGO = 3600
/data/splunk/etc/system/default/props.conf           MAX_DIFF_SECS_HENCE = 604800
/data/splunk/etc/system/default/props.conf           MAX_EVENTS = 256
/data/splunk/etc/system/default/props.conf           MAX_TIMESTAMP_LOOKAHEAD = 128
/data/splunk/etc/system/default/props.conf           MUST_BREAK_AFTER =
/data/splunk/etc/system/default/props.conf           MUST_NOT_BREAK_AFTER =
/data/splunk/etc/system/default/props.conf           MUST_NOT_BREAK_BEFORE =
/data/splunk/etc/system/default/props.conf           SEGMENTATION = indexing
/data/splunk/etc/system/default/props.conf           SEGMENTATION-all = full
/data/splunk/etc/system/default/props.conf           SEGMENTATION-inner = inner
/data/splunk/etc/system/default/props.conf           SEGMENTATION-outer = outer
/data/splunk/etc/system/default/props.conf           SEGMENTATION-raw = none
/data/splunk/etc/system/default/props.conf           SEGMENTATION-standard = standard
/data/splunk/etc/system/default/props.conf           SHOULD_LINEMERGE = True
/data/splunk/etc/system/default/props.conf           TRANSFORMS =
/data/splunk/etc/system/default/props.conf           TRUNCATE = 10000
/data/splunk/etc/system/default/props.conf           detect_trailing_nulls = false
/data/splunk/etc/system/default/props.conf           maxDist = 100
/data/splunk/etc/system/default/props.conf           priority =
/data/splunk/etc/system/default/props.conf           sourcetype =
0 Karma
Highlighted

Re: Why are JSON fields extracted and displayed twice?

Motivator

from where did you took this btool? UF, IDX, SH? check mainly in UF and IDX

------------
Hope I was able to help you. If so, an upvote would be appreciated.
0 Karma
Highlighted

Re: Why are JSON fields extracted and displayed twice?

Path Finder

I checked that on SH.

0 Karma
Highlighted

Re: Why are JSON fields extracted and displayed twice?

Path Finder

Anyone has anymore clues as how to debug this?. I have also run the query on CM, there also I see the duplicate JSON values.

0 Karma
Highlighted

Re: Why are JSON fields extracted and displayed twice?

Engager

Hi,

I also faced a similar issue a while ago.
I believe the JSON rows are getting duplicated in the SH UI.

Possibly, this is due to multiple JSON parsing for the source type due to splunk config file precedence.

Kindly check on the btool configuration to troubleshoot the issue
Use the below command to see the conf for source type.
1)Go to your Splunk bin directory where your app resides.
2) ./splunk btool props list --debug | grep "your source type"
3)See if the JSON conf are coming from a higher precedence file.
4)Set the KVMODE=none and AUTOKV_JSON=false based on this.

Hope this helps!!

dP

0 Karma
Highlighted

Re: Why are JSON fields extracted and displayed twice?

Path Finder

Thanks for your response.
I checked above steps and props are coming/used from where I defined. They are same as what you mentioned in step-4. Still same issue.

$/opt/splunk/bin/splunk btool props list --debug | grep "my_sourcetype"
/data/splunk/etc/apps/my_app/local/props.conf            [my_sourcetype]
0 Karma