topic Re: JSON - Duplicated Fields in Getting Data In

JSON - Duplicated Fields

verbal_666 — Fri, 13 Sep 2019 16:32:22 GMT

Splunk Enterprise 7.0.2

Can't get rid of duplicated fields indexed in a json format. I tryied all combinations, in IDX Env and SH Env, both equals, then different, no way AT ALL.

JSON, very simple:

[
  {
    "name": "Name1",
    "id": 1,
    "age": 20
  }, {
    "name": "Name2",
    "id": 8,
    "age": 30
  }, {
    "name": "Name3",
    "id": 12,
    "age": 40
  }
]

Props IDX and SH, equal,

[JSON]
INDEXED_EXTRACTIONS = json
SHOULD_LINEMERGE = true
NO_BINARY_CHECK = true
CHARSET=UTF-8
KV_MODE = json
AUTO_KV_JSON = true

Results,

I can't "mvdedup" all fields, also because this is a simple test json, then i'll have to index complex with arrays and hundreds fields...

Solutions? It's getting me mad!!!

Thanks.

Re: JSON - Duplicated Fields

verbal_666 — Sun, 15 Sep 2019 11:33:10 GMT

Find a solution.
I used the "preconfigured"

"_json" sourcetype in "/opt/splunk/etc/system/default/props.conf"

[_json]
pulldown_type = true
INDEXED_EXTRACTIONS = json
KV_MODE = none
category = Structured
description = JavaScript Object Notation format. For more information, visit http://json.org/

... ingestion produced no duplicated fields...

Re: JSON - Duplicated Fields

verbal_666 — Sun, 15 Sep 2019 11:38:49 GMT

The only problem is now with much complex json format, like {} formatted,

{
"data": [
    {
        "displayName": "First Name",
        "rank": 1,
        "value": "VALUE"
    },
    {
        "displayName": "Last Name",
        "rank": 2,
        "value": "VALUE"
    },
    {
        "displayName": "Position",
        "rank": 3,
        "value": "VALUE"
    },
    {
        "displayName": "Company Name",
        "rank": 4,
        "value": "VALUE"
    },
    {
        "displayName": "Country",
        "rank": 5,
        "value": "VALUE"
    }
]
}

This is not indexed with multiple events, but single events and multivalue fields...

Re: JSON - Duplicated Fields

verbal_666 — Mon, 16 Sep 2019 20:37:05 GMT

Another strange behaviour.
This is the perfect/right props.conf to index the json, also more complicated (1000 records with multiple arrays),

[my_json]
pulldown_type = true
INDEXED_EXTRACTIONS = json
KV_MODE = none
category = Structured
TRUNCATE=0
JSON_TRIM_BRACES_IN_ARRAY_NAMES=true

NOW, on my test-Environment (single istance idx,sh all),
1) if i let props.conf stay in etc/apps/MY_TA/default/ , fields are duplicated, no truncate (seems the TA works as a simple Forwarder)
2) if i move the props.conf in etc/system/local, json is perfectly indexed (now seems the props is used as Indexer)

...

Re: JSON - Duplicated Fields

verbal_666 — Tue, 17 Sep 2019 17:50:26 GMT

Resolved also the "props path problem", adjusting the "metadata/local.meta" of the TA, setting global.