Getting Data In

JSON - Duplicated Fields

verbal_666
Communicator

Splunk Enterprise 7.0.2

Can't get rid of duplicated fields indexed in a json format. I tryied all combinations, in IDX Env and SH Env, both equals, then different, no way AT ALL.

JSON, very simple:

[
  {
    "name": "Name1",
    "id": 1,
    "age": 20
  }, {
    "name": "Name2",
    "id": 8,
    "age": 30
  }, {
    "name": "Name3",
    "id": 12,
    "age": 40
  }
]

Props IDX and SH, equal,

[JSON]
INDEXED_EXTRACTIONS = json
SHOULD_LINEMERGE = true
NO_BINARY_CHECK = true
CHARSET=UTF-8
KV_MODE = json
AUTO_KV_JSON = true

Results,

alt text

alt text

I can't "mvdedup" all fields, also because this is a simple test json, then i'll have to index complex with arrays and hundreds fields...

Solutions? It's getting me mad!!!

Thanks.

0 Karma

verbal_666
Communicator

Resolved also the "props path problem", adjusting the "metadata/local.meta" of the TA, setting global.

0 Karma

verbal_666
Communicator

Another strange behaviour.
This is the perfect/right props.conf to index the json, also more complicated (1000 records with multiple arrays),

[my_json]
pulldown_type = true
INDEXED_EXTRACTIONS = json
KV_MODE = none
category = Structured
TRUNCATE=0
JSON_TRIM_BRACES_IN_ARRAY_NAMES=true

NOW, on my test-Environment (single istance idx,sh all),
1) if i let props.conf stay in etc/apps/MY_TA/default/ , fields are duplicated, no truncate (seems the TA works as a simple Forwarder)
2) if i move the props.conf in etc/system/local, json is perfectly indexed (now seems the props is used as Indexer)

...

0 Karma

verbal_666
Communicator

The only problem is now with much complex json format, like {} formatted,

{
"data": [
    {
        "displayName": "First Name",
        "rank": 1,
        "value": "VALUE"
    },
    {
        "displayName": "Last Name",
        "rank": 2,
        "value": "VALUE"
    },
    {
        "displayName": "Position",
        "rank": 3,
        "value": "VALUE"
    },
    {
        "displayName": "Company Name",
        "rank": 4,
        "value": "VALUE"
    },
    {
        "displayName": "Country",
        "rank": 5,
        "value": "VALUE"
    }
]
}

This is not indexed with multiple events, but single events and multivalue fields...

0 Karma

verbal_666
Communicator

Find a solution.
I used the "preconfigured"

"_json" sourcetype in "/opt/splunk/etc/system/default/props.conf"

[_json]
pulldown_type = true
INDEXED_EXTRACTIONS = json
KV_MODE = none
category = Structured
description = JavaScript Object Notation format. For more information, visit http://json.org/

... ingestion produced no duplicated fields...

0 Karma
Get Updates on the Splunk Community!

Ready, Set, SOAR: How Utility Apps Can Up Level Your Playbooks!

 WATCH NOW Powering your capabilities has never been so easy with ready-made Splunk® SOAR Utility Apps. Parse ...

DevSecOps: Why You Should Care and How To Get Started

 WATCH NOW In this Tech Talk we will talk about what people mean by DevSecOps and deep dive into the different ...

Introducing Ingest Actions: Filter, Mask, Route, Repeat

WATCH NOW Ingest Actions (IA) is the best new way to easily filter, mask and route your data in Splunk® ...