Bug: Why are there duplicate values with INDEXED_EXTRACTION?

[ololdach]

Hi,

this is a long running issue with splunk creating duplicates as multi-value mv fields when JSON extraction runs at index time and at search time. Especially in a distributed environment it can be mindboggling to find the right set of configurations to finally make it work. Can somebody please give us some details or documentation on how the whole extraction process works internally? I feel that we're all in the "try and error" state of mind and I'd really like to progress to the "knowing what actually happens so that we can cope" stage.

Hint for development: Ideally splunk would be smart enough to realize that if a field already has been extracted at index time, there is no need to do it again. A simple if clause in the code could make the whole configuration issue a lot simpler, speed up search time extractions and make apps in distributed environments more maintainable.

Oliver

[sanjeev543]

I am facing duplicates though I have changed my local.meta file

[test_json]
AUTO_KV_JSON = false
INDEXED_EXTRACTIONS = json
KV_MODE = none

local meta 

[]
access = read : [ * ], write : [ * ]
export = system

Environment is distributed and I am ingesting this data from Search head. and sample data I tried with  is  https://jsonformatter.org/json-editor/a2ec9f 

[divman]

I have a JSON file with .json extension which has a complete one line unstructured json. any events gets added to the json array with the same one line json every 5 minutes.

Gone through multiple responses related to duplicate events for JSON, this is what my configurations looks both on search head and indexer props.conf , but still I can see duplicate events when searching on search head

[dell:boomi:atom]
LINE_BREAKER=(\},)
MUST_BREAK_AFTER=([\},])
SHOULD_LINEMERGE=false
SEDCMD-remove_header=s/({"jmx":\[)//g
SEDCMD-remove_footer=s/(}]})//g
INDEXED_EXTRACTIONS = JSON
KV_MODE = none
AUTO_KV_JSON = false
TIME_PREFIX={"(?=\d+-\d+-\d+T)
TIME_FORMAT=%Y-%m-%dT%H:%M:%S.%3N
MAX_TIMESTAMP_LOOKAHEAD=24
TRUNCATE = 0

[ololdach · Accepted solution]

The Bug/Problem is: A user defined JSON sourcetype that has INDEXED_EXTRACTION=true will result in all fields being displayed as duplicate value mv fields when searched. This happens even if the KV_MODE is set to none for this sourcetype.

We did extensive testing to nail down this issue both on single-instance and distributed environments and it drove us mad, because the one config working here did not work in another seemingly identical environments. After a lot of research it boiled down to a simple visibility issue. Here are our lessons learned:
1. The whole issue is caused by search time artifacts and only the search head configurations need to be changed
2. The props.conf with the sourcetype definition including the KV_MODE=none have to be visible/accessible in the context of the search.
3. When you define the sourcetype inside a TA, separated from the app that does the searches, you need to include the export=system in your local.meta file. In our case, inside the TA, we simply forgot to include export = system in the sourcetype's stanza in ./metadata/default.meta. Once we added the export setting, the duplicated values in our searches were gone.

Still, we consider it a cludge that splunk does not realise that a JSON has already been extracted at index time, wasting additional time to re-extract it at search time. We hope that our ordeal helps others to save time on the subject.

[splunkreal]

Thanks for solution, you're right, export=system was needed on SHC 🙂

[badrinath_itrs]

Hi ,

This is a very good example as how SPLUNK is handling JSON data .

https://www.hurricanelabs.com/blog/splunk-case-study-indexed-extractions-vs-search-time-extractions

One thing to keep in mind, if you use INDEXED_EXTRACTIONS=json, then set KV_MODE=none. If you are not using INDEXED_EXTRACTIONS then use KV_MODE=json .

Hope this is what you are looking for.

[ololdach]

Hi, thanks for your help. We finally got this one right and because there have been several people looking for hints on how to resolve this duplicate value issue, we decided to highlight our answer below.

[divman]

I have a JSON file with .json extension which has a complete one line unstructured json. any events gets added to the json array with the same one line json every 5 minutes.

Gone through multiple responses related to duplicate events for JSON, this is what my configurations looks both on search head and indexer props.conf , but still I can see duplicate events when searching on search head

[dell:boomi:atom]
LINE_BREAKER=(\},)
MUST_BREAK_AFTER=([\},])
SHOULD_LINEMERGE=false
SEDCMD-remove_header=s/({"jmx":\[)//g
SEDCMD-remove_footer=s/(}]})//g
INDEXED_EXTRACTIONS = JSON
KV_MODE = none
AUTO_KV_JSON = false
TIME_PREFIX={"(?=\d+-\d+-\d+T)
TIME_FORMAT=%Y-%m-%dT%H:%M:%S.%3N
MAX_TIMESTAMP_LOOKAHEAD=24
TRUNCATE = 0