Splunk Search

How to find duplicates on multiple fields?

sejiweji
New Member

I have logs with the following three fields:

-category 

-price 

-requestID (unique per entry)

I want to find all requestID's for entries that have BOTH the same category and price within a 1 hour time span.

I started off with this query: 

index=foo component="shop-service" | streamstats count as dupes by category, price
| search dupes> 1

But I cannot seem to calculate the duplicate entries nor tie it to the requestID

 

Labels (4)
Tags (2)
0 Karma

bowesmana
SplunkTrust
SplunkTrust

I assume you are searching a time window longer than 1 hour if you are using streamstats. If you are only searching 60 minutes, then stats will work.

To collect the requestIDs, use values(requestID) in the streamstats command

index=foo component="shop-service" 
| streamstats time_window=1h values(requestID) as requestIDs by category price
| where mvcount(requestIDs) > 1

This will collect all unique requestIDs that have the same category and price and the mvcount() does the > 1 test. 

Note that there are event limitations using streamstats with long time windows, see the docs, so be aware.

 

0 Karma

ITWhisperer
SplunkTrust
SplunkTrust

Try eventstats not streamstats

index=foo component="shop-service" | eventstats count as dupes by category, price
| search dupes> 1
0 Karma

PaulPanther
Motivator
index=foo component="shop-service"
| stats list(request_id) count as dupes by category, price 
| where dupes > 1
0 Karma
Get Updates on the Splunk Community!

Dashboards: Hiding charts while search is being executed and other uses for tokens

There are a couple of features of SimpleXML / Classic dashboards that can be used to enhance the user ...

Splunk Observability Cloud's AI Assistant in Action Series: Explaining Metrics and ...

This is the fourth post in the Splunk Observability Cloud’s AI Assistant in Action series that digs into how ...

Brains, Bytes, and Boston: Learn from the Best at .conf25

When you think of Boston, you might picture colonial charm, world-class universities, or even the crack of a ...