topic Re: Bug: Duplicate values with INDEXED_EXTRACTION? in Getting Data In

Bug: Why are there duplicate values with INDEXED_EXTRACTION?

ololdach — Thu, 19 Jan 2023 18:26:18 GMT

Hi,

this is a long running issue with splunk creating duplicates as multi-value mv fields when JSON extraction runs at index time and at search time. Especially in a distributed environment it can be mindboggling to find the right set of configurations to finally make it work. Can somebody please give us some details or documentation on how the whole extraction process works internally? I feel that we're all in the "try and error" state of mind and I'd really like to progress to the "knowing what actually happens so that we can cope" stage.

Hint for development: Ideally splunk would be smart enough to realize that if a field already has been extracted at index time, there is no need to do it again. A simple if clause in the code could make the whole configuration issue a lot simpler, speed up search time extractions and make apps in distributed environments more maintainable.

Oliver

Re: Bug: Duplicate values with INDEXED_EXTRACTION?

badrinath_itrs — Wed, 30 Sep 2020 02:38:12 GMT

Hi ,

This is a very good example as how SPLUNK is handling JSON data .

https://www.hurricanelabs.com/blog/splunk-case-study-indexed-extractions-vs-search-time-extractions

One thing to keep in mind, if you use INDEXED_EXTRACTIONS=json, then set KV_MODE=none. If you are not using INDEXED_EXTRACTIONS then use KV_MODE=json .

Hope this is what you are looking for.

Re: Bug: Duplicate values with INDEXED_EXTRACTION?

ololdach — Wed, 30 Sep 2020 02:38:15 GMT

The Bug/Problem is: A user defined JSON sourcetype that has INDEXED_EXTRACTION=true will result in all fields being displayed as duplicate value mv fields when searched. This happens even if the KV_MODE is set to none for this sourcetype.

We did extensive testing to nail down this issue both on single-instance and distributed environments and it drove us mad, because the one config working here did not work in another seemingly identical environments. After a lot of research it boiled down to a simple visibility issue. Here are our lessons learned:
1. The whole issue is caused by search time artifacts and only the search head configurations need to be changed
2. The props.conf with the sourcetype definition including the KV_MODE=none have to be visible/accessible in the context of the search.
3. When you define the sourcetype inside a TA, separated from the app that does the searches, you need to include the export=system in your local.meta file. In our case, inside the TA, we simply forgot to include export = system in the sourcetype's stanza in ./metadata/default.meta. Once we added the export setting, the duplicated values in our searches were gone.

Still, we consider it a cludge that splunk does not realise that a JSON has already been extracted at index time, wasting additional time to re-extract it at search time. We hope that our ordeal helps others to save time on the subject.

Re: Bug: Duplicate values with INDEXED_EXTRACTION?

ololdach — Wed, 16 Oct 2019 12:16:51 GMT

Hi, thanks for your help. We finally got this one right and because there have been several people looking for hints on how to resolve this duplicate value issue, we decided to highlight our answer below.

Re: Bug: Duplicate values with INDEXED_EXTRACTION?

divman — Mon, 09 Nov 2020 12:46:57 GMT

I have a JSON file with .json extension which has a complete one line unstructured json. any events gets added to the json array with the same one line json every 5 minutes.

Gone through multiple responses related to duplicate events for JSON, this is what my configurations looks both on search head and indexer props.conf , but still I can see duplicate events when searching on search head

[dell:boomi:atom]
LINE_BREAKER=(\},)
MUST_BREAK_AFTER=([\},])
SHOULD_LINEMERGE=false
SEDCMD-remove_header=s/({"jmx":\[)//g
SEDCMD-remove_footer=s/(}]})//g
INDEXED_EXTRACTIONS = JSON
KV_MODE = none
AUTO_KV_JSON = false
TIME_PREFIX={"(?=\d+-\d+-\d+T)
TIME_FORMAT=%Y-%m-%dT%H:%M:%S.%3N
MAX_TIMESTAMP_LOOKAHEAD=24
TRUNCATE = 0

Re: Bug: Duplicate values with INDEXED_EXTRACTION?

divman — Mon, 09 Nov 2020 12:47:47 GMT

I have a JSON file with .json extension which has a complete one line unstructured json. any events gets added to the json array with the same one line json every 5 minutes.

[dell:boomi:atom]
LINE_BREAKER=(\},)
MUST_BREAK_AFTER=([\},])
SHOULD_LINEMERGE=false
SEDCMD-remove_header=s/({"jmx":\[)//g
SEDCMD-remove_footer=s/(}]})//g
INDEXED_EXTRACTIONS = JSON
KV_MODE = none
AUTO_KV_JSON = false
TIME_PREFIX={"(?=\d+-\d+-\d+T)
TIME_FORMAT=%Y-%m-%dT%H:%M:%S.%3N
MAX_TIMESTAMP_LOOKAHEAD=24
TRUNCATE = 0

Re: Bug: Duplicate values with INDEXED_EXTRACTION?

sanjeev543 — Sun, 09 May 2021 16:50:38 GMT

I am facing duplicates though I have changed my local.meta file

[test_json] AUTO_KV_JSON = false INDEXED_EXTRACTIONS = json KV_MODE = none

local meta

[] access = read : [ * ], write : [ * ] export = system

Environment is distributed and I am ingesting this data from Search head. and sample data I tried with is https://jsonformatter.org/json-editor/a2ec9f

Re: Bug: Duplicate values with INDEXED_EXTRACTION?

splunkreal — Thu, 19 Jan 2023 18:22:59 GMT

Thanks for solution, you're right, export=system was needed on SHC 🙂

Re: Bug: Duplicate values with INDEXED_EXTRACTION?

tomapatan — Tue, 06 Feb 2024 23:37:52 GMT

Experiencing the same issue, can you advise where did you define this setting, please ? We`re using Splunk Cloud, so not sure how to access the local.meta file ?

Re: Bug: Why are there duplicate values with INDEXED_EXTRACTION?

rphillips_splk — Wed, 20 Mar 2024 22:06:59 GMT

the INDEXED_EXTRACTIONS configuration belongs in props.conf of the universal forwarder.

|tstats count where index=* sourcetype=my_json_data by host | stats values(host)

The search above should tell you which hosts need to be looked at where you would remove INDEXED_EXTRACTIONS = json from the SHs and Indexers and move this configuration (INDEXED_EXTRACTIONS = json) to the forwarders props.conf.

Make sure the forwarder inputs.conf for the json source you are ingesting is tagging the data with the appropriate sourcetype, then in props.conf reference that sourcetype stanza for your config:

ie (UF):
inputs.conf

[monitor:///file] sourcetype=foo_json index=bar

props.conf

[foo_json] INDEXED_EXTRACTIONS = json

see:https://docs.splunk.com/Documentation/Splunk/6.5.2/Admin/Configurationparametersandt[…]A.&_ga=2.147263155.568450395.1710801981-1206481253.1693859797

INDEXED_EXTRACTIONS are unique in that they happen in the structured parsing queue of the universal forwarder where usually parsing happens at a HF or indexer if there is no HF.
if you use a HF as the first point of ingest and no UF then you place it there on the HF.

see: https://docs.splunk.com/Documentation/SplunkCloud/latest/Data/Extractfieldsfromfileswithstructureddata

If you have Splunk Cloud Platform and want configure the extraction of fields from structured data, use the Splunk universal forwarder.

Re: Bug: Why are there duplicate values with INDEXED_EXTRACTION?

tomapatan — Thu, 21 Mar 2024 15:53:14 GMT

We`re ingesting data using a REST API call, not a UF, but still experiencing the issue with duplicate values.

We created an app using the Add-on Builder app then deployed it onto one of the HF which ingests and sends the data to Cloud.

Settings on the HF:

KV_MODE = none INDEXED_EXTRACTIONS = json

Any advice would be appreciated.

Thanks,

Toma

Re: Bug: Why are there duplicate values with INDEXED_EXTRACTION?

PickleRick — Thu, 21 Mar 2024 15:39:36 GMT

The KV_MODE (and AUTO_KV_JSON) are options needed on search-heads, not HFs/indexers.

Re: Bug: Why are there duplicate values with INDEXED_EXTRACTION?

tomapatan — Tue, 26 Mar 2024 14:05:13 GMT

Settings on the SH as follows:

AUTO_KV_JSON = false KV_MODE = none

Settings on the HF:

AUTO_KV_JSON = false INDEXED_EXTRACTIONS = json KV_MODE = none

Values are getting duplicated, do you have anymore suggestions for us ?