Splunk Search

How to find duplicates on multiple fields?

sejiweji
New Member

I have logs with the following three fields:

-category 

-price 

-requestID (unique per entry)

I want to find all requestID's for entries that have BOTH the same category and price within a 1 hour time span.

I started off with this query: 

index=foo component="shop-service" | streamstats count as dupes by category, price
| search dupes> 1

But I cannot seem to calculate the duplicate entries nor tie it to the requestID

 

Labels (4)
Tags (2)
0 Karma

bowesmana
SplunkTrust
SplunkTrust

I assume you are searching a time window longer than 1 hour if you are using streamstats. If you are only searching 60 minutes, then stats will work.

To collect the requestIDs, use values(requestID) in the streamstats command

index=foo component="shop-service" 
| streamstats time_window=1h values(requestID) as requestIDs by category price
| where mvcount(requestIDs) > 1

This will collect all unique requestIDs that have the same category and price and the mvcount() does the > 1 test. 

Note that there are event limitations using streamstats with long time windows, see the docs, so be aware.

 

0 Karma

ITWhisperer
SplunkTrust
SplunkTrust

Try eventstats not streamstats

index=foo component="shop-service" | eventstats count as dupes by category, price
| search dupes> 1
0 Karma

PaulPanther
Motivator
index=foo component="shop-service"
| stats list(request_id) count as dupes by category, price 
| where dupes > 1
0 Karma
Get Updates on the Splunk Community!

Splunk Observability Cloud’s AI Assistant in Action Series: Analyzing and ...

This is the second post in our Splunk Observability Cloud’s AI Assistant in Action series, in which we look at ...

Elevate Your Organization with Splunk’s Next Platform Evolution

 Thursday, July 10, 2025  |  11AM PDT / 2PM EDT Whether you're managing complex deployments or looking to ...

Splunk Answers Content Calendar, June Edition

Get ready for this week’s post dedicated to Splunk Dashboards! We're celebrating the power of community by ...