topic Re: Automatic extraction of JSON fields doesn't work and have to use spath in Getting Data In

Automatic extraction of JSON fields doesn't work and have to use spath

Motoko89 — Wed, 02 May 2018 01:04:54 GMT

I have a distributed Splunk deployment and need to index JSON data, 1 object per row. Objects are serialized using NewtonSoft.JSON .NET library. I already configured both INDEXED_EXTRACTIONS = json and KV_MODE = json for my custom source type in props.conf for deployment app of forwarders, indexers, and heads.

Yet, when I search, I still need to specify spath else no result is returned. Our event text length does not exceed 5000 chars (~ 2500 chars) so I don't think the problem here https://answers.splunk.com/answers/177410/why-are-only-some-json-fields-extracted-as-data-in.html affect us. What do I miss here? Any suggestion?

My props.conf

[myblob]
DATETIME_CONFIG = 
INDEXED_EXTRACTIONS = json
KV_MODE = json
AUTO_KV_JSON = true
NO_BINARY_CHECK = true
category = Structured
description = my blob description
disabled = false
TIME_PREFIX=\"EventInfo\.Time\":\s*
TIME_FORMAT=%Y-%m-%dT%H:%M:%S.%7NZ

Re: Automatic extraction of JSON fields doesn't work and have to use spath

ssadanala1 — Tue, 29 Sep 2020 19:23:47 GMT

You cant specify both INDEXED_EXTRACTIONS and KV_MODE in the props .
You need to specify either of one .

Try this

[myblob]
DATETIME_CONFIG =
KV_MODE = json
AUTO_KV_JSON = true
NO_BINARY_CHECK = true
category = Structured
description = my blob description
disabled = false
TIME_PREFIX=\"EventInfo.Time\":\s*
TIME_FORMAT=%Y-%m-%dT%H:%M:%S.%7NZ

Re: Automatic extraction of JSON fields doesn't work and have to use spath

xpac — Tue, 29 Sep 2020 19:19:47 GMT

...and you most likely want to use KV_MODE and not indexed extractions. Remember that in case 9d KV_MODE, this has to go on the search head, not on the indexer. In case of indexed extractions, it has to go on the indexer, and maybe on the search head (not sure about that).

Re: Automatic extraction of JSON fields doesn't work and have to use spath

Motoko89 — Tue, 29 Sep 2020 19:24:01 GMT

So there are 3 places the props.conf go to: the forwarder, the indexer, and the head. Could you advice more specifically which one I should change into what? Do I keep AUTO_KV_JSON at all 3 places?

Re: Automatic extraction of JSON fields doesn't work and have to use spath

MuS — Wed, 02 May 2018 20:25:47 GMT

AUTO_KV_JSON is not needed anywhere, because it defaults to true anyway 😉

AUTO_KV_JSON = [true|false]
* Used for search-time field extractions only.
* Specifies whether to try json extraction automatically.
* Defaults to true.

If you use KV_MODE=json it will only go on the search head. And adding to this, your JSON format must be true and valid JSON otherwise Splunk will not do anything ...

* The 'xml' and 'json' modes will not extract any fields when used on data
  that isn't of the correct format (JSON or XML).

Hope this helps ...

cheers, MuS