JSON fields are extracted twice.
On Universal forwarder (7.0.3) the settings props.conf
are like this
[my_sourcetype]
SHOULD_LINEMERGE=true
LINE_BREAKER=([\r\n]+)
NO_BINARY_CHECK=true
CHARSET=UTF-8
INDEXED_EXTRACTIONS=json
KV_MODE=none
category=Structured
disabled=false
pulldown_type=true
TIMESTAMP_FIELDS=timestamp
On Search Head(7.2.6), tried all combinations of below in props.conf
[my_sourcetype]
INDEXED_EXTRACTIONS=json
KV_MODE=none
AUTO_KV_JSON = false
We ended up doing below which works the way we want i.e. no duplicate json values.
On UF, do NOT define any props.
On Indexers, nothing specific to JSON props, but we had defined props related time field
[my_sourcetype]
NO_BINARY_CHECK=true
CHARSET=UTF-8
MAX_TIMESTAMP_LOOKAHEAD=14000
TIME_PREFIX=timestamp":"?
On SH, do NOT define any props
With this set up, The JSON values are by default extracted in Indexing layer. Because on Indexers, this property is set up in system/default location.
[default]
AUTO_KV_JSON = true
We ended up doing below which works the way we want i.e. no duplicate json values.
On UF, do NOT define any props.
On Indexers, nothing specific to JSON props, but we had defined props related time field
[my_sourcetype]
NO_BINARY_CHECK=true
CHARSET=UTF-8
MAX_TIMESTAMP_LOOKAHEAD=14000
TIME_PREFIX=timestamp":"?
On SH, do NOT define any props
With this set up, The JSON values are by default extracted in Indexing layer. Because on Indexers, this property is set up in system/default location.
[default]
AUTO_KV_JSON = true
Hi,
I also faced a similar issue a while ago.
I believe the JSON rows are getting duplicated in the SH UI.
Possibly, this is due to multiple JSON parsing for the source type due to splunk config file precedence.
Kindly check on the btool configuration to troubleshoot the issue
Use the below command to see the conf for source type.
1)Go to your Splunk bin directory where your app resides.
2) ./splunk btool props list --debug | grep "your source type"
3)See if the JSON conf are coming from a higher precedence file.
4)Set the KV_MODE=none and AUTO_KV_JSON=false based on this.
Hope this helps!!
dP
We ended up doing below which works the way we want i.e. no duplicate json values.
On UF, do NOT define any props.
On Indexers, nothing specific to JSON props, but we had defined props related time field
[my_sourcetype]
NO_BINARY_CHECK=true
CHARSET=UTF-8
MAX_TIMESTAMP_LOOKAHEAD=14000
TIME_PREFIX=timestamp":"?
On SH, do NOT define any props
With this set up, The JSON values are by default extracted in Indexing layer. Because on Indexers, this property is set up in system/default location.
[default]
AUTO_KV_JSON = true
Thanks for your response.
I checked above steps and props are coming/used from where I defined. They are same as what you mentioned in step-4. Still same issue.
$/opt/splunk/bin/splunk btool props list --debug | grep "my_sourcetype"
/data/splunk/etc/apps/my_app/local/props.conf [my_sourcetype]
This is the correct command /opt/splunk/bin/splunk btool props list --debug my_sourcetype
gripping the name of "my_sourcetype" will just show you the sourcetype stanza and not the attribute being applied to it
Anyone has anymore clues as how to debug this?. I have also run the query on CM, there also I see the duplicate JSON values.
What do you mean with "JSON fields are extracted twice."?
Also INDEXED_EXTRACTIONS is use during indexing stage in UFs or IDXs. So unless you are indexing data using you search head, there is no point on this particular atribute being there.
check this. It shows where in the indexing pipeline each atribute is used.
https://wiki.splunk.com/Community:HowIndexingWorks
Correct. I tried below as well on SH.
[my_sourcetype]
KV_MODE=none
AUTO_KV_JSON = false
try to run a btool to check whatever is also being used with you sourcetype
in CLI splunk btool props list --debug my_sourcetype
It shows this
[root@lgpbdus4101 bin]# ./splunk btool props list --debug my_sourcetype
/data/splunk/etc/apps/my_app/local/props.conf [my_sourcetype]
/data/splunk/etc/system/default/props.conf ADD_EXTRA_TIME_FIELDS = True
/data/splunk/etc/system/default/props.conf ANNOTATE_PUNCT = True
/data/splunk/etc/apps/my_app/local/props.conf AUTO_KV_JSON = false
/data/splunk/etc/system/default/props.conf BREAK_ONLY_BEFORE =
/data/splunk/etc/system/default/props.conf BREAK_ONLY_BEFORE_DATE = True
/data/splunk/etc/system/default/props.conf CHARSET = UTF-8
/data/splunk/etc/system/default/props.conf DATETIME_CONFIG = /etc/datetime.xml
/data/splunk/etc/system/default/props.conf DEPTH_LIMIT = 1000
/data/splunk/etc/system/default/props.conf HEADER_MODE =
/data/splunk/etc/apps/my_app/local/props.conf KV_MODE = none
/data/splunk/etc/system/default/props.conf LEARN_MODEL = true
/data/splunk/etc/system/default/props.conf LEARN_SOURCETYPE = true
/data/splunk/etc/system/default/props.conf LINE_BREAKER_LOOKBEHIND = 100
/data/splunk/etc/system/default/props.conf MATCH_LIMIT = 100000
/data/splunk/etc/system/default/props.conf MAX_DAYS_AGO = 2000
/data/splunk/etc/system/default/props.conf MAX_DAYS_HENCE = 2
/data/splunk/etc/system/default/props.conf MAX_DIFF_SECS_AGO = 3600
/data/splunk/etc/system/default/props.conf MAX_DIFF_SECS_HENCE = 604800
/data/splunk/etc/system/default/props.conf MAX_EVENTS = 256
/data/splunk/etc/system/default/props.conf MAX_TIMESTAMP_LOOKAHEAD = 128
/data/splunk/etc/system/default/props.conf MUST_BREAK_AFTER =
/data/splunk/etc/system/default/props.conf MUST_NOT_BREAK_AFTER =
/data/splunk/etc/system/default/props.conf MUST_NOT_BREAK_BEFORE =
/data/splunk/etc/system/default/props.conf SEGMENTATION = indexing
/data/splunk/etc/system/default/props.conf SEGMENTATION-all = full
/data/splunk/etc/system/default/props.conf SEGMENTATION-inner = inner
/data/splunk/etc/system/default/props.conf SEGMENTATION-outer = outer
/data/splunk/etc/system/default/props.conf SEGMENTATION-raw = none
/data/splunk/etc/system/default/props.conf SEGMENTATION-standard = standard
/data/splunk/etc/system/default/props.conf SHOULD_LINEMERGE = True
/data/splunk/etc/system/default/props.conf TRANSFORMS =
/data/splunk/etc/system/default/props.conf TRUNCATE = 10000
/data/splunk/etc/system/default/props.conf detect_trailing_nulls = false
/data/splunk/etc/system/default/props.conf maxDist = 100
/data/splunk/etc/system/default/props.conf priority =
/data/splunk/etc/system/default/props.conf sourcetype =
from where did you took this btool? UF, IDX, SH? check mainly in UF and IDX
I checked that on SH.