topic Re: Why would INDEXED_EXTRACTIONS=JSON in props.conf be resulting in duplicate values? in Getting Data In

Why would INDEXED_EXTRACTIONS=JSON in props.conf be resulting in duplicate values?

pumphreyaw — Wed, 19 Jul 2017 16:49:41 GMT

Using Splunk to analyze bro network transaction data in JSON format. I noticed the stats command and field summary stats would show a count of 2 for unique session ID's, although search results only show one event. After a lot of verification I'm certain my event source does not contain duplicate events.

Thanks to this post: https://answers.splunk.com/answers/223095/why-is-my-sourcetype-configuration-for-json-events.html, I started messing with my JSON settings in props.conf. I thought this would be my fix, but I found the opposite scenario to be true for me...

In short, I'm seeing that using index-time JSON field extractions are resulting in duplicate field values, where search-time JSON field extractions are not.

In props.conf, this produces duplicate values, visible in stats command and field summaries:

INDEXED_EXTRACTIONS=JSON
KV_MODE=none
AUTO_KV_JSON=false

If I disable indexed extractions and use search-time extractions instead, no more duplicate field values:

#INDEXED_EXTRACTIONS=JSON
KV_MODE=json
AUTO_KV_JSON=true

From what I can tell this behavior is different than what others reported in earlier posts. I'm running Splunk 6.6.2 Enterprise on a Debian VM and a 6.6.2 Universal Forwarder on another VM. Maybe there is a deployment client configuration I have wrong somewhere that is causing weird behavior for index-time extractions but no luck so far.

Using search-time extractions seems to work fine, but wondering if anyone is seeing this or if there are any ideas on root cause.

Thanks.

Re: Why would INDEXED_EXTRACTIONS=JSON in props.conf be resulting in duplicate values?

mattymo — Tue, 29 Sep 2020 14:57:11 GMT

Hey pumphreyaw!

It comes down to WHERE you make these changes. If you use INDEXED_EXTRACTIONS, the props.conf needs to be on the UF ( Universal Forwarder VM ), and the KV_MODE=NONE needs to be on the Search Head (aka your Splunk Enterprise VM).

From what I read above, setting the INDEXED_EXTRACTIONS and disabling KV_MODE=JSON should work.

Where did you disable the KV_MODE configs?

Re: Why would INDEXED_EXTRACTIONS=JSON in props.conf be resulting in duplicate values?

pumphreyaw — Wed, 19 Jul 2017 18:37:14 GMT

I think you nailed it. The props.conf file I'm modifying in this case belongs to a deployment app that's getting pushed to the UF, none of which is going to the Search Head. I see I need to split these props settings up accordingly. I'll give that a try. Thanks for the help and quick reply.

Re: Why would INDEXED_EXTRACTIONS=JSON in props.conf be resulting in duplicate values?

mattymo — Wed, 19 Jul 2017 18:43:17 GMT

awesome, I have converted the comment to answer. Let me know if it works!

Re: Why would INDEXED_EXTRACTIONS=JSON in props.conf be resulting in duplicate values?

pumphreyaw — Thu, 20 Jul 2017 12:44:54 GMT

Yep, that worked perfectly. Oversight on my part, just needed to put things in the right place.

Thanks mmodestino!

Re: Why would INDEXED_EXTRACTIONS=JSON in props.conf be resulting in duplicate values?

jperry_intact — Tue, 29 Sep 2020 15:40:11 GMT

I cannot get this to work for the life of me. I can get the json events to only index once if I upload the file and select the sourcetype. If I set it as a monitor input for the same sourcetype and the same files, I get duplicate events. Initially I was getting duplicate events(same event listed twice) and duplicate field extractions(1 field, 2 identical values). Adding INDEXED_EXTRACTIONS = JSON seemed to fix the duplicate field extractions

Its on a single server install on my local machine and I have tried creating the props.conf entry below in both C:\Program Files\Splunk\etc\system\local and C:\Program Files\Splunk\etc\apps\INSERTAPPNAMEHERE\local and no dice.

[FishNPickles]
INDEXED_EXTRACTIONS = JSON
TIMESTAMP_FIELDS = properties.LastUpdateTime
TZ = UTC
AUTO_KV_JSON = false
DATETIME_CONFIG =
KV_MODE = none
SHOULD_LINEMERGE = false
category = Custom
description = PicklesNFish
disabled = false
pulldown_type = true

Is there some secret sauce to this I'm missing? It just straight up ignores the KV_MODE settings and is still indexing my entities twice.

Any direction you could provide would be ultra awesome and greatly appreciated!

Re: Why would INDEXED_EXTRACTIONS=JSON in props.conf be resulting in duplicate values?

jperry_intact — Thu, 07 Sep 2017 14:53:50 GMT

I have apparently done something horrible to my local install. I brought up a new host the your solution works great.

Who knows...

Re: Why would INDEXED_EXTRACTIONS=JSON in props.conf be resulting in duplicate values?

mallempati — Wed, 30 Sep 2020 01:29:44 GMT

hi @mmodestino [Splunk] ♦

By removing the INDEXED_EXTRACTIONS = json from the props.conf on the UF has fixed the issue of duplicates. But it started giving another issue that is sometimes its missing few json event lines.

KV_MODE = none
NO_BINARY_CHECK = true
TIMESTAMP_FIELDS = requests.Time
category = Structured
disabled = false
pulldown_type = true

Any idea how to fix this issue.

Re: Why would INDEXED_EXTRACTIONS=JSON in props.conf be resulting in duplicate values?

joesrepsolc — Tue, 06 Oct 2020 20:46:43 GMT

Any easy to read lists exist of WHERE to use each of these options in the props.conf? I run into this from time to time and its not 100% clear to me WHERE they need to go.

Sometimes it clears says "input time" on this reference (https://docs.splunk.com/Documentation/Splunk/7.2.4/Admin/Propsconf) but other times it doesn't and I'm not sure what that means then.

Any help would be GREAT!!!

Re: Why would INDEXED_EXTRACTIONS=JSON in props.conf be resulting in duplicate values?

Karthikeya — Wed, 15 Jan 2025 12:54:36 GMT

Hi @pumphreyaw , @mattymo

Now I am stuck in same problem. We don't have HF actually. We have deployment server which push apps to our manager and deployer. From there manager will push apps to peers nodes. We have 3 search heads and a deployer.

Where I need to give these configurations to extract json data? Can you please help me step by step?

Re: Why would INDEXED_EXTRACTIONS=JSON in props.conf be resulting in duplicate values?

mattymo — Wed, 15 Jan 2025 14:53:18 GMT

Hey @Karthikeya

as always "it depends" what you mean by "extract json data" and what problem you are trying to solve? Are you seeing duplicate extractions?

This thread talks about indexed extractions settings (in your case it would be needed on the UF) and search time "kv mode" settings (which would be on the Search Head) colliding. "Indexed Extractions" I would not suggest be the first step when working with JSON data. Splunk can extract well formed json at "search time", saving storage and search performance if it is not necessary to store the entire json blob in indexed fields.

So can you clarify what exactly you are doing, or even better post a new question and we can move there?

thanks!

Re: Why would INDEXED_EXTRACTIONS=JSON in props.conf be resulting in duplicate values?

Karthikeya — Wed, 15 Jan 2025 15:15:58 GMT

Hi @mattymo ,

Here is the question link - https://community.splunk.com/t5/Getting-Data-In/Query-to-be-auto-applied/m-p/708893..

Please help me out there.