topic How to find duplicates on multiple fields? in Splunk Search

How to find duplicates on multiple fields?

sejiweji — Mon, 06 Feb 2023 17:56:28 GMT

I have logs with the following three fields:

-category

-price

-requestID (unique per entry)

I want to find all requestID's for entries that have BOTH the same category and price within a 1 hour time span.

I started off with this query:

index=foo component="shop-service" | streamstats count as dupes by category, price
| search dupes> 1

But I cannot seem to calculate the duplicate entries nor tie it to the requestID

PaulPanther — Mon, 06 Feb 2023 15:17:19 GMT

index=foo component="shop-service" | stats list(request_id) count as dupes by category, price | where dupes > 1

ITWhisperer — Mon, 06 Feb 2023 15:23:24 GMT

Try eventstats not streamstats

index=foo component="shop-service" | eventstats count as dupes by category, price | search dupes> 1

bowesmana — Tue, 07 Feb 2023 02:36:19 GMT

I assume you are searching a time window longer than 1 hour if you are using streamstats. If you are only searching 60 minutes, then stats will work.

To collect the requestIDs, use values(requestID) in the streamstats command

index=foo component="shop-service" | streamstats time_window=1h values(requestID) as requestIDs by category price | where mvcount(requestIDs) > 1

This will collect all unique requestIDs that have the same category and price and the mvcount() does the > 1 test.

Note that there are event limitations using streamstats with long time windows, see the docs, so be aware.