Getting Data In

JSON - Duplicated Fields

verbal_666
Builder

Splunk Enterprise 7.0.2

Can't get rid of duplicated fields indexed in a json format. I tryied all combinations, in IDX Env and SH Env, both equals, then different, no way AT ALL.

JSON, very simple:

[
  {
    "name": "Name1",
    "id": 1,
    "age": 20
  }, {
    "name": "Name2",
    "id": 8,
    "age": 30
  }, {
    "name": "Name3",
    "id": 12,
    "age": 40
  }
]

Props IDX and SH, equal,

[JSON]
INDEXED_EXTRACTIONS = json
SHOULD_LINEMERGE = true
NO_BINARY_CHECK = true
CHARSET=UTF-8
KV_MODE = json
AUTO_KV_JSON = true

Results,

alt text

alt text

I can't "mvdedup" all fields, also because this is a simple test json, then i'll have to index complex with arrays and hundreds fields...

Solutions? It's getting me mad!!!

Thanks.

0 Karma

verbal_666
Builder

Resolved also the "props path problem", adjusting the "metadata/local.meta" of the TA, setting global.

0 Karma

verbal_666
Builder

Another strange behaviour.
This is the perfect/right props.conf to index the json, also more complicated (1000 records with multiple arrays),

[my_json]
pulldown_type = true
INDEXED_EXTRACTIONS = json
KV_MODE = none
category = Structured
TRUNCATE=0
JSON_TRIM_BRACES_IN_ARRAY_NAMES=true

NOW, on my test-Environment (single istance idx,sh all),
1) if i let props.conf stay in etc/apps/MY_TA/default/ , fields are duplicated, no truncate (seems the TA works as a simple Forwarder)
2) if i move the props.conf in etc/system/local, json is perfectly indexed (now seems the props is used as Indexer)

...

0 Karma

verbal_666
Builder

The only problem is now with much complex json format, like {} formatted,

{
"data": [
    {
        "displayName": "First Name",
        "rank": 1,
        "value": "VALUE"
    },
    {
        "displayName": "Last Name",
        "rank": 2,
        "value": "VALUE"
    },
    {
        "displayName": "Position",
        "rank": 3,
        "value": "VALUE"
    },
    {
        "displayName": "Company Name",
        "rank": 4,
        "value": "VALUE"
    },
    {
        "displayName": "Country",
        "rank": 5,
        "value": "VALUE"
    }
]
}

This is not indexed with multiple events, but single events and multivalue fields...

0 Karma

verbal_666
Builder

Find a solution.
I used the "preconfigured"

"_json" sourcetype in "/opt/splunk/etc/system/default/props.conf"

[_json]
pulldown_type = true
INDEXED_EXTRACTIONS = json
KV_MODE = none
category = Structured
description = JavaScript Object Notation format. For more information, visit http://json.org/

... ingestion produced no duplicated fields...

0 Karma
Get Updates on the Splunk Community!

Announcing Scheduled Export GA for Dashboard Studio

We're excited to announce the general availability of Scheduled Export for Dashboard Studio. Starting in ...

Extending Observability Content to Splunk Cloud

Watch Now!   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to leverage ...

More Control Over Your Monitoring Costs with Archived Metrics GA in US-AWS!

What if there was a way you could keep all the metrics data you need while saving on storage costs?This is now ...