Splunk Search

How to determine if duplicate logs are sent

balash1979
Path Finder

There was an issue with our Splunk forwarders and it appears our application sent duplicate logs.
I am seeing a sudden spike of log count around a certain time.
Is there a way to know there was, in fact, duplicate logs?
What can I add to my search to find that

index="docker_index" <== This is the search that i am using.

0 Karma
1 Solution

ivanreis
Builder

Hi balash1979, you can run the query below to check your data

Look for offset in the WathedFile component:
index=_internal sourcetype=splunkd component=watchedfile
- checksum fro seekptr didn't match, will re-read entire file
- file too small to check seekcrc
- will begin reading at offset=0 means a file is new(or rolled)
- seeing this twice in other conditions means it is not good

index"docker_index" sourcetype=xyz | convert ctime(_indextime) AS idxtime
| stats count dc(idxtime) as numIndexed, values(source), values(idxtime) by _raw
| where count > 1

View solution in original post

balash1979
Path Finder

great. This works. Is there a way I can use timechart and show between the 2 times (that I use for the search), when the count > 1 was higher or lower

0 Karma

ivanreis
Builder

try this:
| timechart count dc(idxtime) as numIndexed, values(source), values(idxtime) by _raw where count > 1

0 Karma

ivanreis
Builder

Hi balash1979, you can run the query below to check your data

Look for offset in the WathedFile component:
index=_internal sourcetype=splunkd component=watchedfile
- checksum fro seekptr didn't match, will re-read entire file
- file too small to check seekcrc
- will begin reading at offset=0 means a file is new(or rolled)
- seeing this twice in other conditions means it is not good

index"docker_index" sourcetype=xyz | convert ctime(_indextime) AS idxtime
| stats count dc(idxtime) as numIndexed, values(source), values(idxtime) by _raw
| where count > 1

Get Updates on the Splunk Community!

BSides Splunk 2022 - The Call for Papers is now Open!

TLDR; Main Site: https://bsidessplunk.com CFP Site: https://bsidessplunk.com/cfp CFP Opens: December 15th, ...

Sending Metrics to Splunk Enterprise With the OpenTelemetry Collector

This blog post is part of an ongoing series on OpenTelemetry. The OpenTelemetry project is the second largest ...

What's New in Splunk Cloud Platform 9.0.2208?!

Howdy!  We are happy to share the newest updates in Splunk Cloud Platform 9.0.2208! Analysts can benefit ...