topic Re: Duplicate data (_raw) in Getting Data In

Why am I seeing Duplicate data (_raw)?

VijaySrrie — Thu, 17 Aug 2023 18:01:30 GMT

Hi,

I could see duplicate data in splunk by using below query

index="indexname"
| stats count by _raw
| where count >1

I checked 10 different indexes out of which 8 indexes are having duplicate logs

Re: Duplicate data (_raw)

gcusello — Thu, 17 Aug 2023 06:44:42 GMT

Hi @VijaySrrie,

you have to analyze your data sources: some of them aren't correctly configurated.

The most common misconfigurations are the following:

installed forwarders in an active/active cluster,
receiving syslog using two syslog servers but without a Load Balancer,
use the crcSal = <SOURCE> with logs that rotate or tar the old files.

the first job should be, using the following search, undertanding of what are the duplicated data source:

index="indexname" | stats values(sourcetype) AS sourcetype count by _raw | where count>1

in this way you have the list of duplicated sourcetypes and you can focus your analyis on those sourcetypes.

Then, if the duplicated sourcetypes come from a cluster or syslogs, you should analize your architecture to understand eventual duplications.

Ciao.

Giuseppe

Re: Duplicate data (_raw)

ITWhisperer — Thu, 17 Aug 2023 07:12:52 GMT

Other duplications can come from how your logs are managed at source, e.g. are they replicated across multiple servers, or are they renamed and picked up multiple times. Try these to find these sorts of occurrences.

index="indexname" | stats values(source) AS source count by _raw | where count>1

index="indexname" | stats values(host) AS host count by _raw | where count>1

Re: Duplicate data (_raw)

PickleRick — Thu, 17 Aug 2023 11:51:57 GMT

Apart from the issues already covered by @ITWhisperer and @gcusello there is also the possibility of duplication caused by useACK parameter, communication problems and retrasnsmission. Check your _internal log for "possible duplication".