Data from a particular source is extracted in duplicate when searched from "Searching and reporting"?

[joemiller]

This is a single server Splunk deployment. I am indexing Duo MFA logs using the official splunk app. In the "Searching and reporting" app, when I use the table command to view that data, each field is a multivalue field with the value duplicated. When I try the same search using the Duo app instead of "Searching and reporting", the fields are extracted only once as expected, not duplicated. For example...

When I use the table command on this data in "Searching and reporting":

email
user@example.com
user@example.com

When I use the table command on this data in the "Duo" app:

email
user@example.com

So this problem appears to be limited to the "Searching and reporting" app. But I'm not finding any configuration specific to "Searching and reporting" related to this app/source. For example, there is nothing in SPLUNK/etc/apps/search/local/ props.conf or transforms.conf that would affect this source.

The current configuration according to the btool is coming from SPLUNK/etc/apps/duo_splunkapp/default/props.conf and is:

[source::duo]
INDEXED_EXTRACTIONS = json
KV_MODE = none

I also tried changing the config to this:

INDEXED_EXTRACTIONS = none
AUTO_KV_JSON = true
KV_MODE = json

but that just resulted in neither the "Searching and reporting" app nor the "Duo" app having extractions for this data. How do I fix this so the "Searching and reporting" app has a single set of extractions and not duplicates?

[joemiller]

Update: despite the btool not showing anything relevant in SPLUNK/etc/apps/search/local/props.conf, I added the following lines to that file:

[source::duo]
AUTO_KV_JSON = false
KV_MODE = none

and this seems to have fixed my problem. So now my question is why? Why weren't the lines in SPLUNK/etc/apps/duo_splunkapp/default/props.conf and (duo_splunkapp/local/props.conf) taking effect even though there were no contradicting lines in a conf file with greater precedence?

[yeahnah · Accepted solution]

Hi @joemiller 

The issue is using INDEXED_EXTRACTION = json (forwarder/heavy forwarder config) and having KV_MODE = json (search head configuration)

https://docs.splunk.com/Documentation/Splunk/9.0.3/Admin/Propsconf#Structured_Data_Header_Extraction...

The INDEXED_EXTRACTION setting means the fields are indexed at ingestion time.  If this is done then KV_MODE = none (default is auto) should be set on the search head.  Also ensure this configuration is globally shared for the sourcetype/source.

In this case the KV_MODE = none had only been shared in the duo_splunkapp app space, i.e. the search and reporting app does not see this config.  To share its config globally: Apps > Manage apps >  duo_splunkapp > Permissions > All apps (system) 

Personally, I would avoid using INDEXED_EXTRACTION and then just let the Splunk search head KV_MODE = auto do all the work at search time - this is considered best practise.  Sadly, a lot of third Splunk party apps come with an INDEXED_EXTRACTION enabled and deployed in the default directory.  The last time I tired, the only way to disable an INDEXED_EXTRACTION was to remove the line from props.conf in the default folder, i.e. it cannot be disabled in the local directory props.conf with an INDEXED_EXTRACTION = none entry (admittedly, I've not tried this for a while).   Not being able to disable it in a local folder means any upgrade of this app in the future will enable it again - unless manually removed again.

Anyway, bit off topic, but you were on the right track with the settings you were playing with.

Hope this helps

[joemiller]

Thank you! Argh, so the permissions setting on the app was the reason why my changes weren't taking effect. Thanks for clearing that up for me. And that's confusing about the INDEXED_EXTRACTION not being able to be modified from the local props.conf. I understand why search time extraction mightbe preferable, but I don't want to create a setup that will break when there's an app update either.