topic Re: KV_MODE=json sometimes skips a particular JSON field? in Splunk Search

KV_MODE=json sometimes skips a particular JSON field?

zanglang — Wed, 28 Aug 2019 10:20:55 GMT

We have a log file with multiple lines of JSON similar to this:

{ "foo": "bar","foo1":"foo2","userEmail":"foo@bar.com"}
{ "foo": "bar","foo1":"foo2","userEmail":"foo1@bar.com"}
{ "foo": "bar","foo1":"foo2","userEmail":"foo2@bar.com"}

And search-time extraction works fine for almost all of the fields... except one! Oddly, around 7-8% of all logs do not have userEmail automatically extracted as checked in the Event Coverage, even when I've manually defined it in props.conf. This was verified with the queries:

index=foo | search userEmail=*
index=foo | search NOT userEmail=*

Events are sent from a forwarder with this props.conf:

[foo]
NO_BINARY_CHECK = true
SHOULD_LINEMERGE = false
TIME_FORMAT = %Y%m%d%H%M%S%3N
TIME_PREFIX = \"timestamp\":\"
TZ = UTC
KV_MODE = json
disabled = false
TRUNCATE = 0

I added these on the search head earlier today to force search-time extraction for userEmail, but didn't work, even when I verified the regex catches all emails in Splunk Web:

[foo]
EXTRACT-userEmail = "userEmail":"(?P<userEmail>[^"]+)
KV_MODE = json

Any idea why this might happen?

Re: KV_MODE=json sometimes skips a particular JSON field?

wenthold — Wed, 28 Aug 2019 20:08:51 GMT

Have you tried btool - if you're running on Linux I would try something like

$SPLUNK_HOME/bin/splunk btool props list --debug | grep "userEmail"

Field aliasing, calculated fields, and lookups are all performed at search time after KV extractions, maybe there's an errant config that's overwriting your field.

Re: KV_MODE=json sometimes skips a particular JSON field?

woodcock — Sun, 01 Sep 2019 20:24:02 GMT

Open a support case.

Re: KV_MODE=json sometimes skips a particular JSON field?

jhollfelder_spl — Wed, 30 Sep 2020 01:59:02 GMT

I'm experiencing the same thing.

I have JSON formatted data from the NetApp ONTAP add-on that contains a pool_id field. If I search the correct index and sourcetype and add " |extract" or "| spath", pool_id gets extracted correctly otherwise it extracts what appears to be all other fields except for this one. Scratching head...

Re: KV_MODE=json sometimes skips a particular JSON field?

zanglang — Thu, 05 Sep 2019 06:07:38 GMT

Turns out the issue was caused by an autolookup with an outdated CSV lookup file where userEmail was one of the autolookup fields. Updated the CSV fixed my issue.

In this case, it appears that all userEmails that did exist in the lookup table would autolookup and extract correctly, and userEmails that were not present in the CSV would fail autolookup and for some reason also broke field extraction.

Re: KV_MODE=json sometimes skips a particular JSON field?

richgalloway — Thu, 05 Sep 2019 12:51:10 GMT

@zanglang If your problem is resolved, please accept an answer to help future readers.

Re: KV_MODE=json sometimes skips a particular JSON field?

jhollfelder_spl — Wed, 30 Sep 2020 02:04:42 GMT

So it turns out that my problem was caused by a FIELDALIAS setting that was setting a field that actually existed in the data with a field that didn't exist. If I had run btool as suggested by @wenthold, I would have found this much faster.

It turns out that if the props.conf add-on had used "ASNEW" instead of "AS" in the FIELDALIAS definition, Splunk would have kept the field extraction it found in the data rather than overwrite it with a field that didn't exist. The update I made to local/props.conf for that add-on was:
FIELDALIAS-array_id = uuid ASNEW array_id