Getting Data In

How do I stop getting duplicate entries of JSON data from an API feed?

Contributor

I installed the Duo Security App that uses the API to download events in the JSON format. The data is collected and when I perform a search, the events look correct but the data in fields is doubled. Looking through Answers, there are several responses that suggest that props.conf be modified on the UF (adding KV_MODE = none) to prevent both index time extractions and search time extractions. Since the data is not be forwarded, which props.conf do I need to modify? I put KV_MODE = none in props.conf under the apps\duo_splunkapp\local but there was no change.

Regards,

Scott

1 Solution

SplunkTrust
SplunkTrust

Did you add that in props.conf on your indexer or search head ? It should be on the search head.

View solution in original post

0 Karma

New Member

I'm a little late to the party, but for those with a distributed deployment using a Heavy Forwarder for collecting and forwarding the Duo logs, here is what I had to do to fix this issue:

I don't have the Duo app on the indexers.
On the heavy forward, I kept the default configurations in props.conf.
On the search head cluster in props.conf, I set:

[source::duo]
KV_MODE = none
AUTO_KV_JSON = false

 You could unset INDEXED_EXTRACTIONS but since these are dedicated search heads, it isn't really necessary. 

That's it.

Note that if you want this to work outside of the Duo app context (i.e. not have the duplicate field extractions when searching Duo logs in the Search app) then you need to set the Duo apps permissions to global.

0 Karma

SplunkTrust
SplunkTrust

Did you add that in props.conf on your indexer or search head ? It should be on the search head.

View solution in original post

0 Karma

Contributor

I have a single instance.

0 Karma

SplunkTrust
SplunkTrust

In that case make sure INDEXED_EXTRACTIONS=JSON and KV_MODE = json are both unset.

0 Karma

Contributor

David, how would I unset INDEXED_EXTRACTIONS? According to the props.conf doc, there is no "none" parameter only CSV|TSV|PSV|W3C|JSON|HEC.

The props.conf in apps\duo_splunkapp\default is configured
[source::duo]
INDEXED_EXTRACTIONS = json
KV_MODE = none
TIME_PREFIX = timestamp
TIME_FORMAT = %s

The props.conf in apps\duo_splunkapp\local is configured
[source::duo]
AUTO_KV_JSON = false

0 Karma

SplunkTrust
SplunkTrust

Simply set it as follows :

[source::duo]
INDEXED_EXTRACTIONS =

Then it will be unset 🙂

0 Karma

Contributor

As nothing is this easy in Splunk, I added the line with my fingers crossed. Restarted Splunk and did a test.

Each field is now showing the correct number of values.

Thank you for the help.

0 Karma

Path Finder

@scottrunyon - Can you post your full \local\props.conf and \default\props.conf ? We have the same problem, but when we unset indexed_extractions and kv_mode, we just get a giant chunk of raw json. I'm not sure where else our settings differ.

0 Karma

Observer

@peterm30 - Try adding SHOULD_LINEMERGE = false to your [source::duo] stanza as well.

0 Karma

Contributor

peterm30, I would like to help you out but we have moved away from DUO and no longer use the DUO app in Splunk.

0 Karma

SplunkTrust
SplunkTrust

hahah, yeah, with index time configs and search time configs things can get confusing very easily. I'm glad I could help ! 🙂

0 Karma
State of Splunk Careers

Access the Splunk Careers Report to see real data that shows how Splunk mastery increases your value and job satisfaction.

Find out what your skills are worth!