Splunk Search

Find duplicate events for a pattern that occurred in the same timestamp

gbehl
New Member

My requirement is to find duplicate events for a pattern that occurred in the same 'second' of timestamp after stripping the millisecond value.

queries that I tried but didn't give me 100% success:

search_pattern | timechart span=1s count | where count >1

search_pattern | timechart span=1s count | where count >1 | table _time, _raw

Not sure if 'eventcount summarize=false' or 'eventstats' would be of any help here.

P.S. I've recently started on splunk hence my knowledge is limited but I can work with pointers and do hit n trial approach.

Any pointers are appreciated.

Tags (1)
0 Karma

woodcock
Esteemed Legend

Like this:

search_pattern | bin span=1s _time | eventstats count AS duplicate_count BY _raw | search duplicate_count > 1
0 Karma

somesoni2
SplunkTrust
SplunkTrust

Give this a try (would be very resource intensive so run for small time ranges)

search_pattern | bucket span=1s _time | stats count first(_raw) as raw by _time | where count>1 | rename raw as _raw
0 Karma

gbehl
New Member

It works fine except the fact that it picks the first row and ignores the other rows. After running this query, I realized that there are some exceptions in data where I would need all the _raw rows which I can get by doing an extra click. I can live with that for now.

Thanks a lot!

0 Karma

somesoni2
SplunkTrust
SplunkTrust

This query is written to collect first row (as we assume they're all same, may be different millisecond). If you want all _raw data in the result, simply replace first(_raw) with values(_raw) (to get unique raw events) OR list(_raw) to get all raw events.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...