Getting Data In

How do I stop getting duplicate entries of JSON data from an API feed?

scottrunyon
Contributor

I installed the Duo Security App that uses the API to download events in the JSON format. The data is collected and when I perform a search, the events look correct but the data in fields is doubled. Looking through Answers, there are several responses that suggest that props.conf be modified on the UF (adding KV_MODE = none) to prevent both index time extractions and search time extractions. Since the data is not be forwarded, which props.conf do I need to modify? I put KV_MODE = none in props.conf under the apps\duo_splunkapp\local but there was no change.

Regards,

Scott

1 Solution

DavidHourani
Super Champion

Did you add that in props.conf on your indexer or search head ? It should be on the search head.

View solution in original post

PBGrantham
New Member

I'm a little late to the party, but for those with a distributed deployment using a Heavy Forwarder for collecting and forwarding the Duo logs, here is what I had to do to fix this issue:

I don't have the Duo app on the indexers.
On the heavy forward, I kept the default configurations in props.conf.
On the search head cluster in props.conf, I set:

[source::duo]
KV_MODE = none
AUTO_KV_JSON = false

 You could unset INDEXED_EXTRACTIONS but since these are dedicated search heads, it isn't really necessary. 

That's it.

Note that if you want this to work outside of the Duo app context (i.e. not have the duplicate field extractions when searching Duo logs in the Search app) then you need to set the Duo apps permissions to global.

0 Karma

DavidHourani
Super Champion

Did you add that in props.conf on your indexer or search head ? It should be on the search head.

View solution in original post

scottrunyon
Contributor

I have a single instance.

0 Karma

DavidHourani
Super Champion

In that case make sure INDEXED_EXTRACTIONS=JSON and KV_MODE = json are both unset.

0 Karma

scottrunyon
Contributor

David, how would I unset INDEXED_EXTRACTIONS? According to the props.conf doc, there is no "none" parameter only CSV|TSV|PSV|W3C|JSON|HEC.

The props.conf in apps\duo_splunkapp\default is configured
[source::duo]
INDEXED_EXTRACTIONS = json
KV_MODE = none
TIME_PREFIX = timestamp
TIME_FORMAT = %s

The props.conf in apps\duo_splunkapp\local is configured
[source::duo]
AUTO_KV_JSON = false

0 Karma

DavidHourani
Super Champion

Simply set it as follows :

[source::duo]
INDEXED_EXTRACTIONS =

Then it will be unset 🙂

0 Karma

scottrunyon
Contributor

As nothing is this easy in Splunk, I added the line with my fingers crossed. Restarted Splunk and did a test.

Each field is now showing the correct number of values.

Thank you for the help.

0 Karma

peterm30
Path Finder

@scottrunyon - Can you post your full \local\props.conf and \default\props.conf ? We have the same problem, but when we unset indexed_extractions and kv_mode, we just get a giant chunk of raw json. I'm not sure where else our settings differ.

0 Karma

Mradyfist
Observer

@peterm30 - Try adding SHOULD_LINEMERGE = false to your [source::duo] stanza as well.

0 Karma

scottrunyon
Contributor

peterm30, I would like to help you out but we have moved away from DUO and no longer use the DUO app in Splunk.

0 Karma

DavidHourani
Super Champion

hahah, yeah, with index time configs and search time configs things can get confusing very easily. I'm glad I could help ! 🙂

0 Karma
.conf21 Now Fully Virtual!
Register for FREE Today!

We've made .conf21 totally virtual and totally FREE! Our completely online experience will run from 10/19 through 10/20 with some additional events, too!