Getting Data In

JSON - Duplicated Fields

verbal_666
Builder

Splunk Enterprise 7.0.2

Can't get rid of duplicated fields indexed in a json format. I tryied all combinations, in IDX Env and SH Env, both equals, then different, no way AT ALL.

JSON, very simple:

[
  {
    "name": "Name1",
    "id": 1,
    "age": 20
  }, {
    "name": "Name2",
    "id": 8,
    "age": 30
  }, {
    "name": "Name3",
    "id": 12,
    "age": 40
  }
]

Props IDX and SH, equal,

[JSON]
INDEXED_EXTRACTIONS = json
SHOULD_LINEMERGE = true
NO_BINARY_CHECK = true
CHARSET=UTF-8
KV_MODE = json
AUTO_KV_JSON = true

Results,

alt text

alt text

I can't "mvdedup" all fields, also because this is a simple test json, then i'll have to index complex with arrays and hundreds fields...

Solutions? It's getting me mad!!!

Thanks.

0 Karma

verbal_666
Builder

Resolved also the "props path problem", adjusting the "metadata/local.meta" of the TA, setting global.

0 Karma

verbal_666
Builder

Another strange behaviour.
This is the perfect/right props.conf to index the json, also more complicated (1000 records with multiple arrays),

[my_json]
pulldown_type = true
INDEXED_EXTRACTIONS = json
KV_MODE = none
category = Structured
TRUNCATE=0
JSON_TRIM_BRACES_IN_ARRAY_NAMES=true

NOW, on my test-Environment (single istance idx,sh all),
1) if i let props.conf stay in etc/apps/MY_TA/default/ , fields are duplicated, no truncate (seems the TA works as a simple Forwarder)
2) if i move the props.conf in etc/system/local, json is perfectly indexed (now seems the props is used as Indexer)

...

0 Karma

verbal_666
Builder

The only problem is now with much complex json format, like {} formatted,

{
"data": [
    {
        "displayName": "First Name",
        "rank": 1,
        "value": "VALUE"
    },
    {
        "displayName": "Last Name",
        "rank": 2,
        "value": "VALUE"
    },
    {
        "displayName": "Position",
        "rank": 3,
        "value": "VALUE"
    },
    {
        "displayName": "Company Name",
        "rank": 4,
        "value": "VALUE"
    },
    {
        "displayName": "Country",
        "rank": 5,
        "value": "VALUE"
    }
]
}

This is not indexed with multiple events, but single events and multivalue fields...

0 Karma

verbal_666
Builder

Find a solution.
I used the "preconfigured"

"_json" sourcetype in "/opt/splunk/etc/system/default/props.conf"

[_json]
pulldown_type = true
INDEXED_EXTRACTIONS = json
KV_MODE = none
category = Structured
description = JavaScript Object Notation format. For more information, visit http://json.org/

... ingestion produced no duplicated fields...

0 Karma
Get Updates on the Splunk Community!

Technical Workshop Series: Splunk Data Management and SPL2 | Register here!

Hey, Splunk Community! Ready to take your data management skills to the next level? Join us for a 3-part ...

Spotting Financial Fraud in the Haystack: A Guide to Behavioral Analytics with Splunk

In today's digital financial ecosystem, security teams face an unprecedented challenge. The sheer volume of ...

Solve Problems Faster with New, Smarter AI and Integrations in Splunk Observability

Solve Problems Faster with New, Smarter AI and Integrations in Splunk Observability As businesses scale ...