Getting Data In

Show only duplicated fields

kmattern
Builder

I have customers who upload sets of files every day. The upload is done automatically. Sometimes there will be a hitch in the system and one or more of the file set will be uploaded multiple times. The file names all have the term _seq_ followed by a sequence number. So part of the customer events will look like this:

abcdef_seq_1
abcdef_seq_2
abcdef_seq_2
abcdef_seq_3
abcdef_seq_4

I only want to show only the duplicated upload files, in this case abcdef_seq_2. It shouldn't be that hard but I'm busting my head. What am I missing?

Ultimately I need to put this into a data model for a Pivot.

0 Karma
1 Solution

kmattern
Builder

I think I finally figured it out. This search returns only those IIS events that have duplicate cs_uri-query fields.

sourcetype="iis" cs_uri_query="*_seq*"  
| stats first(cs_uri_query) as DupFile, first(cs_username) as Customer, count(cs_uri_query) AS Duplicates by cs_uri_query  
| where Duplicates>1 
| table Customer, DupFile, Duplicates

View solution in original post

kmattern
Builder

I think I finally figured it out. This search returns only those IIS events that have duplicate cs_uri-query fields.

sourcetype="iis" cs_uri_query="*_seq*"  
| stats first(cs_uri_query) as DupFile, first(cs_username) as Customer, count(cs_uri_query) AS Duplicates by cs_uri_query  
| where Duplicates>1 
| table Customer, DupFile, Duplicates

yannK
Splunk Employee
Splunk Employee

ps : please mark your question as answered with the left checkbox to accept your own answer 🙂

0 Karma

yannK
Splunk Employee
Splunk Employee

this is the good method.

to find a dulpicate field
* | stats count by myfield | where count>1

to look at the whole events
* | stats count by _raw | where count>1

somesoni2
Revered Legend

In splunk, do you see duplicate data for the files uploaded multiple times?

0 Karma
Get Updates on the Splunk Community!

What’s new on Splunk Lantern in August

This month’s Splunk Lantern update gives you the low-down on all of the articles we’ve published over the past ...

Welcome to the Future of Data Search & Exploration

You have more data coming at you than ever before. Over the next five years, the total amount of digital data ...

This Week's Community Digest - Splunk Community Happenings [8.3.22]

Get the latest news and updates from the Splunk Community here! News From Splunk Answers ✍️ Splunk Answers is ...