Splunk Search

How to determine if duplicate logs are sent

balash1979
Path Finder

There was an issue with our Splunk forwarders and it appears our application sent duplicate logs.
I am seeing a sudden spike of log count around a certain time.
Is there a way to know there was, in fact, duplicate logs?
What can I add to my search to find that

index="docker_index" <== This is the search that i am using.

0 Karma
1 Solution

ivanreis
Builder

Hi balash1979, you can run the query below to check your data

Look for offset in the WathedFile component:
index=_internal sourcetype=splunkd component=watchedfile
- checksum fro seekptr didn't match, will re-read entire file
- file too small to check seekcrc
- will begin reading at offset=0 means a file is new(or rolled)
- seeing this twice in other conditions means it is not good

index"docker_index" sourcetype=xyz | convert ctime(_indextime) AS idxtime
| stats count dc(idxtime) as numIndexed, values(source), values(idxtime) by _raw
| where count > 1

View solution in original post

balash1979
Path Finder

great. This works. Is there a way I can use timechart and show between the 2 times (that I use for the search), when the count > 1 was higher or lower

0 Karma

ivanreis
Builder

try this:
| timechart count dc(idxtime) as numIndexed, values(source), values(idxtime) by _raw where count > 1

0 Karma

ivanreis
Builder

Hi balash1979, you can run the query below to check your data

Look for offset in the WathedFile component:
index=_internal sourcetype=splunkd component=watchedfile
- checksum fro seekptr didn't match, will re-read entire file
- file too small to check seekcrc
- will begin reading at offset=0 means a file is new(or rolled)
- seeing this twice in other conditions means it is not good

index"docker_index" sourcetype=xyz | convert ctime(_indextime) AS idxtime
| stats count dc(idxtime) as numIndexed, values(source), values(idxtime) by _raw
| where count > 1

Get Updates on the Splunk Community!

Improve Your Security Posture

Watch NowImprove Your Security PostureCustomers are at the center of everything we do at Splunk and security ...

Maximize the Value from Microsoft Defender with Splunk

 Watch NowJoin Splunk and Sens Consulting for this Security Edition Tech TalkWho should attend:  Security ...

This Week's Community Digest - Splunk Community Happenings [6.27.22]

Get the latest news and updates from the Splunk Community here! News From Splunk Answers ✍️ Splunk Answers is ...