Splunk Enterprise 7.0.2
Can't get rid of duplicated fields indexed in a json format. I tryied all combinations, in IDX Env and SH Env, both equals, then different, no way AT ALL.
JSON, very simple:
[
{
"name": "Name1",
"id": 1,
"age": 20
}, {
"name": "Name2",
"id": 8,
"age": 30
}, {
"name": "Name3",
"id": 12,
"age": 40
}
]
Props IDX and SH, equal,
[JSON]
INDEXED_EXTRACTIONS = json
SHOULD_LINEMERGE = true
NO_BINARY_CHECK = true
CHARSET=UTF-8
KV_MODE = json
AUTO_KV_JSON = true
Results,
I can't "mvdedup" all fields, also because this is a simple test json, then i'll have to index complex with arrays and hundreds fields...
Solutions? It's getting me mad!!!
Thanks.
Resolved also the "props path problem", adjusting the "metadata/local.meta" of the TA, setting global.
Another strange behaviour.
This is the perfect/right props.conf to index the json, also more complicated (1000 records with multiple arrays),
[my_json]
pulldown_type = true
INDEXED_EXTRACTIONS = json
KV_MODE = none
category = Structured
TRUNCATE=0
JSON_TRIM_BRACES_IN_ARRAY_NAMES=true
NOW, on my test-Environment (single istance idx,sh all),
1) if i let props.conf stay in etc/apps/MY_TA/default/ , fields are duplicated, no truncate (seems the TA works as a simple Forwarder)
2) if i move the props.conf in etc/system/local, json is perfectly indexed (now seems the props is used as Indexer)
...
The only problem is now with much complex json format, like {} formatted,
{
"data": [
{
"displayName": "First Name",
"rank": 1,
"value": "VALUE"
},
{
"displayName": "Last Name",
"rank": 2,
"value": "VALUE"
},
{
"displayName": "Position",
"rank": 3,
"value": "VALUE"
},
{
"displayName": "Company Name",
"rank": 4,
"value": "VALUE"
},
{
"displayName": "Country",
"rank": 5,
"value": "VALUE"
}
]
}
This is not indexed with multiple events, but single events and multivalue fields...
Find a solution.
I used the "preconfigured"
"_json" sourcetype in "/opt/splunk/etc/system/default/props.conf"
[_json]
pulldown_type = true
INDEXED_EXTRACTIONS = json
KV_MODE = none
category = Structured
description = JavaScript Object Notation format. For more information, visit http://json.org/
... ingestion produced no duplicated fields...