Getting Data In

JSON - Duplicated Fields

verbal_666
Builder

Splunk Enterprise 7.0.2

Can't get rid of duplicated fields indexed in a json format. I tryied all combinations, in IDX Env and SH Env, both equals, then different, no way AT ALL.

JSON, very simple:

[
  {
    "name": "Name1",
    "id": 1,
    "age": 20
  }, {
    "name": "Name2",
    "id": 8,
    "age": 30
  }, {
    "name": "Name3",
    "id": 12,
    "age": 40
  }
]

Props IDX and SH, equal,

[JSON]
INDEXED_EXTRACTIONS = json
SHOULD_LINEMERGE = true
NO_BINARY_CHECK = true
CHARSET=UTF-8
KV_MODE = json
AUTO_KV_JSON = true

Results,

alt text

alt text

I can't "mvdedup" all fields, also because this is a simple test json, then i'll have to index complex with arrays and hundreds fields...

Solutions? It's getting me mad!!!

Thanks.

0 Karma

verbal_666
Builder

Resolved also the "props path problem", adjusting the "metadata/local.meta" of the TA, setting global.

0 Karma

verbal_666
Builder

Another strange behaviour.
This is the perfect/right props.conf to index the json, also more complicated (1000 records with multiple arrays),

[my_json]
pulldown_type = true
INDEXED_EXTRACTIONS = json
KV_MODE = none
category = Structured
TRUNCATE=0
JSON_TRIM_BRACES_IN_ARRAY_NAMES=true

NOW, on my test-Environment (single istance idx,sh all),
1) if i let props.conf stay in etc/apps/MY_TA/default/ , fields are duplicated, no truncate (seems the TA works as a simple Forwarder)
2) if i move the props.conf in etc/system/local, json is perfectly indexed (now seems the props is used as Indexer)

...

0 Karma

verbal_666
Builder

The only problem is now with much complex json format, like {} formatted,

{
"data": [
    {
        "displayName": "First Name",
        "rank": 1,
        "value": "VALUE"
    },
    {
        "displayName": "Last Name",
        "rank": 2,
        "value": "VALUE"
    },
    {
        "displayName": "Position",
        "rank": 3,
        "value": "VALUE"
    },
    {
        "displayName": "Company Name",
        "rank": 4,
        "value": "VALUE"
    },
    {
        "displayName": "Country",
        "rank": 5,
        "value": "VALUE"
    }
]
}

This is not indexed with multiple events, but single events and multivalue fields...

0 Karma

verbal_666
Builder

Find a solution.
I used the "preconfigured"

"_json" sourcetype in "/opt/splunk/etc/system/default/props.conf"

[_json]
pulldown_type = true
INDEXED_EXTRACTIONS = json
KV_MODE = none
category = Structured
description = JavaScript Object Notation format. For more information, visit http://json.org/

... ingestion produced no duplicated fields...

0 Karma
Got questions? Get answers!

Join the Splunk Community Slack to learn, troubleshoot, and make connections with fellow Splunk practitioners in real time!

Meet up IRL or virtually!

Join Splunk User Groups to connect and learn in-person by region or remotely by topic or industry.

Get Updates on the Splunk Community!

Mile High Learning with Splunk University, Denver, Colorado

If Denver is known for its mile-high elevation, Splunk University is about to raise the bar on technical ...

IT Service Intelligence 5.0 Series: Your Guide to the June Launch

We are excited to announce the June release of Splunk IT Service Intelligence (ITSI) 5.0. This update ...

Agent Mode Engaged! Enchaining Agentic Operations with Splunk AI Assistant 2.0

    Are you ready to transform how your team handles complex data requests? We invite you to our upcoming ...