Hi,
this is a long running issue with splunk creating duplicates as multi-value mv fields when JSON extraction runs at index time and at search time. Especially in a distributed environment it can be mindboggling to find the right set of configurations to finally make it work. Can somebody please give us some details or documentation on how the whole extraction process works internally? I feel that we're all in the "try and error" state of mind and I'd really like to progress to the "knowing what actually happens so that we can cope" stage.
Hint for development: Ideally splunk would be smart enough to realize that if a field already has been extracted at index time, there is no need to do it again. A simple if clause in the code could make the whole configuration issue a lot simpler, speed up search time extractions and make apps in distributed environments more maintainable.
Oliver
The Bug/Problem is: A user defined JSON sourcetype that has INDEXED_EXTRACTION=true will result in all fields being displayed as duplicate value mv fields when searched. This happens even if the KV_MODE is set to none for this sourcetype.
We did extensive testing to nail down this issue both on single-instance and distributed environments and it drove us mad, because the one config working here did not work in another seemingly identical environments. After a lot of research it boiled down to a simple visibility issue. Here are our lessons learned:
1. The whole issue is caused by search time artifacts and only the search head configurations need to be changed
2. The props.conf with the sourcetype definition including the KV_MODE=none have to be visible/accessible in the context of the search.
3. When you define the sourcetype inside a TA, separated from the app that does the searches, you need to include the export=system in your local.meta file. In our case, inside the TA, we simply forgot to include export = system in the sourcetype's stanza in ./metadata/default.meta. Once we added the export setting, the duplicated values in our searches were gone.
Still, we consider it a cludge that splunk does not realise that a JSON has already been extracted at index time, wasting additional time to re-extract it at search time. We hope that our ordeal helps others to save time on the subject.
|tstats count where index=* sourcetype=my_json_data by host | stats values(host)
The search above should tell you which hosts need to be looked at where you would remove INDEXED_EXTRACTIONS = json from the SHs and Indexers and move this configuration (INDEXED_EXTRACTIONS = json) to the forwarders props.conf.
Make sure the forwarder inputs.conf for the json source you are ingesting is tagging the data with the appropriate sourcetype, then in props.conf reference that sourcetype stanza for your config:
ie (UF):
inputs.conf
[monitor:///file]
sourcetype=foo_json
index=bar
props.conf
[foo_json]
INDEXED_EXTRACTIONS = json
We`re ingesting data using a REST API call, not a UF, but still experiencing the issue with duplicate values.
We created an app using the Add-on Builder app then deployed it onto one of the HF which ingests and sends the data to Cloud.
Settings on the HF:
KV_MODE = none
INDEXED_EXTRACTIONS = json
Any advice would be appreciated.
Thanks,
Toma
The KV_MODE (and AUTO_KV_JSON) are options needed on search-heads, not HFs/indexers.
Settings on the SH as follows:
AUTO_KV_JSON = false
KV_MODE = none
Settings on the HF:
AUTO_KV_JSON = false
INDEXED_EXTRACTIONS = json
KV_MODE = none
Values are getting duplicated, do you have anymore suggestions for us ?
I am facing duplicates though I have changed my local.meta file
[test_json]
AUTO_KV_JSON = false
INDEXED_EXTRACTIONS = json
KV_MODE = none
local meta
[]
access = read : [ * ], write : [ * ]
export = system
Environment is distributed and I am ingesting this data from Search head. and sample data I tried with is https://jsonformatter.org/json-editor/a2ec9f
I have a JSON file with .json extension which has a complete one line unstructured json. any events gets added to the json array with the same one line json every 5 minutes.
Gone through multiple responses related to duplicate events for JSON, this is what my configurations looks both on search head and indexer props.conf , but still I can see duplicate events when searching on search head
[dell:boomi:atom] LINE_BREAKER=(\},) MUST_BREAK_AFTER=([\},]) SHOULD_LINEMERGE=false SEDCMD-remove_header=s/({"jmx":\[)//g SEDCMD-remove_footer=s/(}]})//g INDEXED_EXTRACTIONS = JSON KV_MODE = none AUTO_KV_JSON = false TIME_PREFIX={"(?=\d+-\d+-\d+T) TIME_FORMAT=%Y-%m-%dT%H:%M:%S.%3N MAX_TIMESTAMP_LOOKAHEAD=24 TRUNCATE = 0
The Bug/Problem is: A user defined JSON sourcetype that has INDEXED_EXTRACTION=true will result in all fields being displayed as duplicate value mv fields when searched. This happens even if the KV_MODE is set to none for this sourcetype.
We did extensive testing to nail down this issue both on single-instance and distributed environments and it drove us mad, because the one config working here did not work in another seemingly identical environments. After a lot of research it boiled down to a simple visibility issue. Here are our lessons learned:
1. The whole issue is caused by search time artifacts and only the search head configurations need to be changed
2. The props.conf with the sourcetype definition including the KV_MODE=none have to be visible/accessible in the context of the search.
3. When you define the sourcetype inside a TA, separated from the app that does the searches, you need to include the export=system in your local.meta file. In our case, inside the TA, we simply forgot to include export = system in the sourcetype's stanza in ./metadata/default.meta. Once we added the export setting, the duplicated values in our searches were gone.
Still, we consider it a cludge that splunk does not realise that a JSON has already been extracted at index time, wasting additional time to re-extract it at search time. We hope that our ordeal helps others to save time on the subject.
Thanks for solution, you're right, export=system was needed on SHC 🙂
Experiencing the same issue, can you advise where did you define this setting, please ? We`re using Splunk Cloud, so not sure how to access the local.meta file ?
Hi ,
This is a very good example as how SPLUNK is handling JSON data .
https://www.hurricanelabs.com/blog/splunk-case-study-indexed-extractions-vs-search-time-extractions
One thing to keep in mind, if you use INDEXED_EXTRACTIONS=json, then set KV_MODE=none. If you are not using INDEXED_EXTRACTIONS then use KV_MODE=json .
Hope this is what you are looking for.
Hi, thanks for your help. We finally got this one right and because there have been several people looking for hints on how to resolve this duplicate value issue, we decided to highlight our answer below.
I have a JSON file with .json extension which has a complete one line unstructured json. any events gets added to the json array with the same one line json every 5 minutes.
Gone through multiple responses related to duplicate events for JSON, this is what my configurations looks both on search head and indexer props.conf , but still I can see duplicate events when searching on search head
[dell:boomi:atom] LINE_BREAKER=(\},) MUST_BREAK_AFTER=([\},]) SHOULD_LINEMERGE=false SEDCMD-remove_header=s/({"jmx":\[)//g SEDCMD-remove_footer=s/(}]})//g INDEXED_EXTRACTIONS = JSON KV_MODE = none AUTO_KV_JSON = false TIME_PREFIX={"(?=\d+-\d+-\d+T) TIME_FORMAT=%Y-%m-%dT%H:%M:%S.%3N MAX_TIMESTAMP_LOOKAHEAD=24 TRUNCATE = 0