Duplicate entries in splunk search

[krishna_11]

Hi Splunk experts

I need one help, the splunk search is giving me duplicate entries when I do a search. I have made sure that there are no duplicate events and I have also used dedup in my search. Still it gives me duplicates. Need your help. See the attached image.

Could you please let me know what could be the issue?

Thanks and best regards

Krishna

[alonsocaio]

Hi @krishna_11 , 

Which is the original format of the log entries? Is it JSON or XML?

Maybe you should validate if the sourcetype contains both INDEXED_EXTRACTIONS and KV_MODE set to JSON/XML. If both of them are set, try removing one of them, such as INDEXED_EXTRACTIONS=JSON and KV_MODE=None.

[krishna_11]

Hi @alonsocaio 

The original format of the log entries is JSON.

The sourcetype contains only INDEXED_EXTRACTIONS is set to JSON and I have not set KV_MODE at all.

Are there any other ideas? 

Best regards

Krishna

[alonsocaio]

Are you able to share your sourcetype configs?

[krishna_11]

Hi @alonsocaio 

Here is my source type config:

[source::...ta-audit-logs-ingester*.log*]
sourcetype = taauditlogsingester:log

[source::...ta_audit_logs_ingester*.log*]
sourcetype = taauditlogsingester:log

[Audit-Logs-Source]
INDEXED_EXTRACTIONS = json
SHOULD_LINEMERGE = 0
category = Splunk App Add-on Builder
pulldown_type = 1

Thank you so much for your help. Greatly appreciated.

Thanks and best regards

Krishna

[alonsocaio · Accepted solution]

Hi @krishna_11 

I have tested a simple json file with this sourcetype settings:

[test:json]
INDEXED_EXTRACTIONS = json
SHOULD_LINEMERGE = 0
pulldown_type = 1

And It returned me duplicated values.

When I added KV_MODE = None to the sourcetype, It parsed the json correctly.

[test:json]
INDEXED_EXTRACTIONS = json
SHOULD_LINEMERGE = 0
KV_MODE = None
pulldown_type = 1

I would suggest you to test using the KV_MODE option to validate if It works for you.

[krishna_11]

Hi @alonsocaio 

I have been testing and it looks like your solution worked like a charm. Thank you so much for this amazing solution. Greatly appreciate it. I have been trying for quite some time and could not find any solution. 

You are a lifesaver 🙂

-Krishna