topic Re: Duplicate Extracted Fields (ingest through HEC) in Splunk Search

Duplicate Extracted Fields (ingest through HEC)

mrstrozy — Wed, 30 Sep 2020 02:35:43 GMT

Hi, I am seeing duplicate extractions for events in my Splunk instance. To give a background, I have a couple forwarders (which are mostly not used), an indexer cluster in which each indexer is running the Splunk HEC and the HEC is sitting behind a load balancer, I have one index cluster master and finally I have a search head cluster.

There is not a props.conf file on the search heads.

Currently the props.conf lives in the /opt/splunk/etc/system/local/ directory on the indexers. Here is a sample config:

[mycustomsourcetype]
DATETIME_CONFIG =
INDEXED_EXTRACTIONS = json
KV_MODE = none
AUTO_KV_JSON = false
LINE_BREAKER = ([\r\n]+)
NO_BINARY_CHECK = true
TZ = UTC
category = Structured
description = ticket stuff
disabled = false
TRUNCATE = 0
pulldown_type = true

Considering the only props.conf that I believe should be dictating extractions is on the indexers, I'm very confused about how to fix this.

Any help would be greatly appreciated. Thanks!

Re: Duplicate Extracted Fields (ingest through HEC)

harsmarvania57 — Tue, 22 Oct 2019 23:17:35 GMT

Hi,

Please configure below props.conf on Search Head.

[mycustomsourcetype]
KV_MODE = none

Re: Duplicate Extracted Fields (ingest through HEC)

mrstrozy — Wed, 23 Oct 2019 18:42:23 GMT

The part I forgot to mention was that the events ingested prior to the architecture change did not have duplicate extractions which added to the confusion. I noticed that the replication and search factors were not being met and so I explicitly added to the master server.conf clustering section with "replication_factor = 3" (though I thought this was the case by default). After doing so the replication and search factors were then met and all ingested events from that point on did not have duplicate field extractions.

I am pretty confused by this so anyone with more advanced knowledge please feel free to chime in. I would love to fully understand why this was the case.

Re: Duplicate Extracted Fields (ingest through HEC)

woodcock — Fri, 25 Oct 2019 23:32:49 GMT

Set this on your Search Head:

props.conf

[mycustomsourcetype]
KV_MODE = none
AUTO_KV_JSON = false

Re: Duplicate Extracted Fields (ingest through HEC)

ruman_splunk — Wed, 30 Sep 2020 05:34:36 GMT

I needed both KV_MODE and AUTO_KV_JSON, thanks @woodcock!