topic Duplicate entries in splunk search in Splunk Search

Duplicate entries in splunk search

krishna_11 — Fri, 30 Oct 2020 17:45:10 GMT

Hi Splunk experts

I need one help, the splunk search is giving me duplicate entries when I do a search. I have made sure that there are no duplicate events and I have also used dedup in my search. Still it gives me duplicates. Need your help. See the attached image.

Could you please let me know what could be the issue?

Thanks and best regards

Krishna

Re: Duplicate entries in splunk search

alonsocaio — Fri, 30 Oct 2020 18:55:32 GMT

Hi @krishna_11 ,

Which is the original format of the log entries? Is it JSON or XML?

Maybe you should validate if the sourcetype contains both INDEXED_EXTRACTIONS and KV_MODE set to JSON/XML. If both of them are set, try removing one of them, such as INDEXED_EXTRACTIONS=JSON and KV_MODE=None.

Re: Duplicate entries in splunk search

krishna_11 — Fri, 30 Oct 2020 21:26:50 GMT

Hi @alonsocaio

The original format of the log entries is JSON.

The sourcetype contains only INDEXED_EXTRACTIONS is set to JSON and I have not set KV_MODE at all.

Are there any other ideas?

Best regards

Krishna

Re: Duplicate entries in splunk search

alonsocaio — Sat, 31 Oct 2020 17:12:41 GMT

Are you able to share your sourcetype configs?

Re: Duplicate entries in splunk search

krishna_11 — Sat, 31 Oct 2020 17:35:23 GMT

Hi @alonsocaio

Here is my source type config:

[source::...ta-audit-logs-ingester*.log*]
sourcetype = taauditlogsingester:log

[source::...ta_audit_logs_ingester*.log*]
sourcetype = taauditlogsingester:log

[Audit-Logs-Source]
INDEXED_EXTRACTIONS = json
SHOULD_LINEMERGE = 0
category = Splunk App Add-on Builder
pulldown_type = 1

Thank you so much for your help. Greatly appreciated.

Thanks and best regards

Krishna

Re: Duplicate entries in splunk search

alonsocaio — Sun, 01 Nov 2020 00:54:13 GMT

Hi @krishna_11

I have tested a simple json file with this sourcetype settings:

[test:json] INDEXED_EXTRACTIONS = json SHOULD_LINEMERGE = 0 pulldown_type = 1

And It returned me duplicated values.

When I added KV_MODE = None to the sourcetype, It parsed the json correctly.

[test:json] INDEXED_EXTRACTIONS = json SHOULD_LINEMERGE = 0 KV_MODE = None pulldown_type = 1

I would suggest you to test using the KV_MODE option to validate if It works for you.

Re: Duplicate entries in splunk search

krishna_11 — Mon, 02 Nov 2020 19:30:20 GMT

Hi @alonsocaio

I have been testing and it looks like your solution worked like a charm. Thank you so much for this amazing solution. Greatly appreciate it. I have been trying for quite some time and could not find any solution.

You are a lifesaver 🙂

-Krishna