Getting Data In

Bug: Why are there duplicate values with INDEXED_EXTRACTION?

ololdach
Builder

Hi,

this is a long running issue with splunk creating duplicates as multi-value mv fields when JSON extraction runs at index time and at search time. Especially in a distributed environment it can be mindboggling to find the right set of configurations to finally make it work. Can somebody please give us some details or documentation on how the whole extraction process works internally? I feel that we're all in the "try and error" state of mind and I'd really like to progress to the "knowing what actually happens so that we can cope" stage.

Hint for development: Ideally splunk would be smart enough to realize that if a field already has been extracted at index time, there is no need to do it again. A simple if clause in the code could make the whole configuration issue a lot simpler, speed up search time extractions and make apps in distributed environments more maintainable.

Oliver

Labels (1)
0 Karma
1 Solution

ololdach
Builder

The Bug/Problem is: A user defined JSON sourcetype that has INDEXED_EXTRACTION=true will result in all fields being displayed as duplicate value mv fields when searched. This happens even if the KV_MODE is set to none for this sourcetype.

We did extensive testing to nail down this issue both on single-instance and distributed environments and it drove us mad, because the one config working here did not work in another seemingly identical environments. After a lot of research it boiled down to a simple visibility issue. Here are our lessons learned:
1. The whole issue is caused by search time artifacts and only the search head configurations need to be changed
2. The props.conf with the sourcetype definition including the KV_MODE=none have to be visible/accessible in the context of the search.
3. When you define the sourcetype inside a TA, separated from the app that does the searches, you need to include the export=system in your local.meta file. In our case, inside the TA, we simply forgot to include export = system in the sourcetype's stanza in ./metadata/default.meta. Once we added the export setting, the duplicated values in our searches were gone.

Still, we consider it a cludge that splunk does not realise that a JSON has already been extracted at index time, wasting additional time to re-extract it at search time. We hope that our ordeal helps others to save time on the subject.

View solution in original post

rphillips_splk
Splunk Employee
Splunk Employee
the INDEXED_EXTRACTIONS configuration belongs in props.conf of the universal forwarder.

 

|tstats count where index=* sourcetype=my_json_data by host | stats values(host)


The search above should tell you which hosts need to be looked at where you would remove INDEXED_EXTRACTIONS = json from the SHs and Indexers and move this configuration (INDEXED_EXTRACTIONS = json) to the forwarders props.conf.

Make sure the forwarder inputs.conf for the json source you are ingesting is tagging the data with the appropriate sourcetype, then in props.conf reference that sourcetype stanza for your config:

ie (UF):
inputs.conf

[monitor:///file]
sourcetype=foo_json
index=bar


props.conf

[foo_json]
INDEXED_EXTRACTIONS = json

 

 
see:https://docs.splunk.com/Documentation/Splunk/6.5.2/Admin/Configurationparametersandt[…]A.&_ga=2.1472...


INDEXED_EXTRACTIONS are unique in that they happen in the structured parsing queue of the universal forwarder where usually parsing happens at a HF or indexer if there is no HF.
if you use a HF as the first point of ingest and no UF then you place it there on the HF.

see: https://docs.splunk.com/Documentation/SplunkCloud/latest/Data/Extractfieldsfromfileswithstructuredda...


If you have Splunk Cloud Platform and want configure the extraction of fields from structured data, use the Splunk universal forwarder.
 
0 Karma

tomapatan
Communicator

We`re ingesting data using a REST API call, not a UF, but still experiencing the issue with duplicate values.

We created an app using the Add-on Builder app then deployed it onto one of the HF which ingests and sends the data to Cloud.

Settings on the HF:

KV_MODE = none
INDEXED_EXTRACTIONS = json 

Any advice would be appreciated.

Thanks,

Toma

0 Karma

PickleRick
SplunkTrust
SplunkTrust

The KV_MODE (and AUTO_KV_JSON) are options needed on search-heads, not HFs/indexers.

0 Karma

tomapatan
Communicator

Settings on the SH as follows:

AUTO_KV_JSON = false
KV_MODE = none

 

Settings on the HF:

AUTO_KV_JSON = false
INDEXED_EXTRACTIONS = json
KV_MODE = none

 

Values are getting duplicated, do you have anymore suggestions for us ?

0 Karma

sanjeev543
Communicator

I am facing duplicates though I have changed my local.meta file

[test_json]
AUTO_KV_JSON = false
INDEXED_EXTRACTIONS = json
KV_MODE = none

local meta 

[]
access = read : [ * ], write : [ * ]
export = system


Environment is distributed and I am ingesting this data from Search head. and sample data I tried with  is  https://jsonformatter.org/json-editor/a2ec9f 

 

0 Karma

divman
Observer

I have a JSON file with .json extension which has a complete one line unstructured json. any events gets added to the json array with the same one line json every 5 minutes.

Gone through multiple responses related to duplicate events for JSON, this is what my configurations looks both on search head and indexer props.conf , but still I can see duplicate events when searching on search head

 

[dell:boomi:atom]
LINE_BREAKER=(\},)
MUST_BREAK_AFTER=([\},])
SHOULD_LINEMERGE=false
SEDCMD-remove_header=s/({"jmx":\[)//g
SEDCMD-remove_footer=s/(}]})//g
INDEXED_EXTRACTIONS = JSON
KV_MODE = none
AUTO_KV_JSON = false
TIME_PREFIX={"(?=\d+-\d+-\d+T)
TIME_FORMAT=%Y-%m-%dT%H:%M:%S.%3N
MAX_TIMESTAMP_LOOKAHEAD=24
TRUNCATE = 0
0 Karma

ololdach
Builder

The Bug/Problem is: A user defined JSON sourcetype that has INDEXED_EXTRACTION=true will result in all fields being displayed as duplicate value mv fields when searched. This happens even if the KV_MODE is set to none for this sourcetype.

We did extensive testing to nail down this issue both on single-instance and distributed environments and it drove us mad, because the one config working here did not work in another seemingly identical environments. After a lot of research it boiled down to a simple visibility issue. Here are our lessons learned:
1. The whole issue is caused by search time artifacts and only the search head configurations need to be changed
2. The props.conf with the sourcetype definition including the KV_MODE=none have to be visible/accessible in the context of the search.
3. When you define the sourcetype inside a TA, separated from the app that does the searches, you need to include the export=system in your local.meta file. In our case, inside the TA, we simply forgot to include export = system in the sourcetype's stanza in ./metadata/default.meta. Once we added the export setting, the duplicated values in our searches were gone.

Still, we consider it a cludge that splunk does not realise that a JSON has already been extracted at index time, wasting additional time to re-extract it at search time. We hope that our ordeal helps others to save time on the subject.

splunkreal
Motivator

Thanks for solution, you're right, export=system was needed on SHC 🙂

* If this helps, please upvote or accept solution 🙂 *
0 Karma

tomapatan
Communicator

Experiencing the same issue, can you advise where did you define this setting, please ? We`re using Splunk Cloud, so not sure how to access the local.meta file ?

badrinath_itrs
Communicator

Hi ,

This is a very good example as how SPLUNK is handling JSON data .

https://www.hurricanelabs.com/blog/splunk-case-study-indexed-extractions-vs-search-time-extractions

One thing to keep in mind, if you use INDEXED_EXTRACTIONS=json, then set KV_MODE=none. If you are not using INDEXED_EXTRACTIONS then use KV_MODE=json .

Hope this is what you are looking for.

ololdach
Builder

Hi, thanks for your help. We finally got this one right and because there have been several people looking for hints on how to resolve this duplicate value issue, we decided to highlight our answer below.

0 Karma

divman
Observer

I have a JSON file with .json extension which has a complete one line unstructured json. any events gets added to the json array with the same one line json every 5 minutes.

Gone through multiple responses related to duplicate events for JSON, this is what my configurations looks both on search head and indexer props.conf , but still I can see duplicate events when searching on search head

[dell:boomi:atom]
LINE_BREAKER=(\},)
MUST_BREAK_AFTER=([\},])
SHOULD_LINEMERGE=false
SEDCMD-remove_header=s/({"jmx":\[)//g
SEDCMD-remove_footer=s/(}]})//g
INDEXED_EXTRACTIONS = JSON
KV_MODE = none
AUTO_KV_JSON = false
TIME_PREFIX={"(?=\d+-\d+-\d+T)
TIME_FORMAT=%Y-%m-%dT%H:%M:%S.%3N
MAX_TIMESTAMP_LOOKAHEAD=24
TRUNCATE = 0
0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...