topic Re: How do I stop getting duplicate entries of JSON data from an API feed? in Getting Data In

How do I stop getting duplicate entries of JSON data from an API feed?

scottrunyon — Wed, 30 Sep 2020 01:43:41 GMT

I installed the Duo Security App that uses the API to download events in the JSON format. The data is collected and when I perform a search, the events look correct but the data in fields is doubled. Looking through Answers, there are several responses that suggest that props.conf be modified on the UF (adding KV_MODE = none) to prevent both index time extractions and search time extractions. Since the data is not be forwarded, which props.conf do I need to modify? I put KV_MODE = none in props.conf under the apps\duo_splunkapp\local but there was no change.

Regards,

Scott

Re: How do I stop getting duplicate entries of JSON data from an API feed?

DavidHourani — Wed, 14 Aug 2019 13:37:11 GMT

Did you add that in props.conf on your indexer or search head ? It should be on the search head.

Re: How do I stop getting duplicate entries of JSON data from an API feed?

scottrunyon — Wed, 14 Aug 2019 14:13:39 GMT

I have a single instance.

Re: How do I stop getting duplicate entries of JSON data from an API feed?

DavidHourani — Wed, 14 Aug 2019 14:36:14 GMT

In that case make sure INDEXED_EXTRACTIONS=JSON and KV_MODE = json are both unset.

Re: How do I stop getting duplicate entries of JSON data from an API feed?

scottrunyon — Wed, 30 Sep 2020 01:43:53 GMT

David, how would I unset INDEXED_EXTRACTIONS? According to the props.conf doc, there is no "none" parameter only CSV|TSV|PSV|W3C|JSON|HEC.

The props.conf in apps\duo_splunkapp\default is configured
[source::duo]
INDEXED_EXTRACTIONS = json
KV_MODE = none
TIME_PREFIX = timestamp
TIME_FORMAT = %s

The props.conf in apps\duo_splunkapp\local is configured
[source::duo]
AUTO_KV_JSON = false

Re: How do I stop getting duplicate entries of JSON data from an API feed?

DavidHourani — Wed, 14 Aug 2019 18:07:33 GMT

Simply set it as follows :

[source::duo]
INDEXED_EXTRACTIONS =

Then it will be unset 🙂

Re: How do I stop getting duplicate entries of JSON data from an API feed?

scottrunyon — Wed, 14 Aug 2019 18:26:48 GMT

As nothing is this easy in Splunk, I added the line with my fingers crossed. Restarted Splunk and did a test.

Each field is now showing the correct number of values.

Thank you for the help.

Re: How do I stop getting duplicate entries of JSON data from an API feed?

DavidHourani — Wed, 14 Aug 2019 18:33:38 GMT

hahah, yeah, with index time configs and search time configs things can get confusing very easily. I'm glad I could help ! 🙂

Re: How do I stop getting duplicate entries of JSON data from an API feed?

peterm30 — Wed, 30 Sep 2020 03:40:40 GMT

@scottrunyon - Can you post your full \local\props.conf and \default\props.conf ? We have the same problem, but when we unset indexed_extractions and kv_mode, we just get a giant chunk of raw json. I'm not sure where else our settings differ.

Re: How do I stop getting duplicate entries of JSON data from an API feed?

scottrunyon — Fri, 17 Jan 2020 16:53:04 GMT

peterm30, I would like to help you out but we have moved away from DUO and no longer use the DUO app in Splunk.

Re: How do I stop getting duplicate entries of JSON data from an API feed?

Mradyfist — Thu, 09 Jul 2020 19:45:47 GMT

@peterm30 - Try adding SHOULD_LINEMERGE = false to your [source::duo] stanza as well.

Re: How do I stop getting duplicate entries of JSON data from an API feed?

PBGrantham — Fri, 09 Oct 2020 20:18:52 GMT

I'm a little late to the party, but for those with a distributed deployment using a Heavy Forwarder for collecting and forwarding the Duo logs, here is what I had to do to fix this issue:

I don't have the Duo app on the indexers.
On the heavy forward, I kept the default configurations in props.conf.
On the search head cluster in props.conf, I set:

[source::duo] KV_MODE = none AUTO_KV_JSON = false

You could unset INDEXED_EXTRACTIONS but since these are dedicated search heads, it isn't really necessary.

That's it.

Note that if you want this to work outside of the Duo app context (i.e. not have the duplicate field extractions when searching Duo logs in the Search app) then you need to set the Duo apps permissions to global.