How do I stop getting duplicate entries of JSON data from an API feed?

[scottrunyon]

I installed the Duo Security App that uses the API to download events in the JSON format. The data is collected and when I perform a search, the events look correct but the data in fields is doubled. Looking through Answers, there are several responses that suggest that props.conf be modified on the UF (adding KV_MODE = none) to prevent both index time extractions and search time extractions. Since the data is not be forwarded, which props.conf do I need to modify? I put KV_MODE = none in props.conf under the apps\duo_splunkapp\local but there was no change.

Regards,

Scott

[PBGrantham]

I'm a little late to the party, but for those with a distributed deployment using a Heavy Forwarder for collecting and forwarding the Duo logs, here is what I had to do to fix this issue:

I don't have the Duo app on the indexers.
On the heavy forward, I kept the default configurations in props.conf.
On the search head cluster in props.conf, I set:

[source::duo]
KV_MODE = none
AUTO_KV_JSON = false

 You could unset INDEXED_EXTRACTIONS but since these are dedicated search heads, it isn't really necessary. 

That's it.

Note that if you want this to work outside of the Duo app context (i.e. not have the duplicate field extractions when searching Duo logs in the Search app) then you need to set the Duo apps permissions to global.

[DavidHourani · Accepted solution]

Did you add that in props.conf on your indexer or search head ? It should be on the search head.

[scottrunyon]

I have a single instance.

[DavidHourani]

In that case make sure INDEXED_EXTRACTIONS=JSON and KV_MODE = json are both unset.

[scottrunyon]

David, how would I unset INDEXED_EXTRACTIONS? According to the props.conf doc, there is no "none" parameter only CSV|TSV|PSV|W3C|JSON|HEC.

The props.conf in apps\duo_splunkapp\default is configured
[source::duo]
INDEXED_EXTRACTIONS = json
KV_MODE = none
TIME_PREFIX = timestamp
TIME_FORMAT = %s

The props.conf in apps\duo_splunkapp\local is configured
[source::duo]
AUTO_KV_JSON = false

[DavidHourani]

Simply set it as follows :

[source::duo]
INDEXED_EXTRACTIONS =

Then it will be unset 🙂

[scottrunyon]

As nothing is this easy in Splunk, I added the line with my fingers crossed. Restarted Splunk and did a test.

Each field is now showing the correct number of values.

Thank you for the help.

[peterm30]

@scottrunyon - Can you post your full \local\props.conf and \default\props.conf ? We have the same problem, but when we unset indexed_extractions and kv_mode, we just get a giant chunk of raw json. I'm not sure where else our settings differ.

[Mradyfist]

@peterm30 - Try adding SHOULD_LINEMERGE = false to your [source::duo] stanza as well.

[scottrunyon]

peterm30, I would like to help you out but we have moved away from DUO and no longer use the DUO app in Splunk.

[DavidHourani]

hahah, yeah, with index time configs and search time configs things can get confusing very easily. I'm glad I could help ! 🙂