Splunk Enterprise

Eval command questions- How many duplicated events do we have?

sqauyyu123
New Member

How many duplicated events we have? Percent of duplicated events?

Difference between duplicated and unique events.?

Labels (1)
0 Karma

johnhuang
Motivator

It depends on how you define or considered to be a duplicate event. Assuming you're talking about the _raw event data you can try this query:

 

<base_search>
| eval event_hash=MD5(nt_host)
| stats count AS event_ct by event_hash
| appendpipe [| stats sum(event_ct) AS total_ct]
| appendpipe [| stats count AS unique_ct]
| where event_ct>1 OR ISNULL(event_ct)
| appendpipe [| where ISNOTNULL(event_ct) | stats count AS events_with_dupe_ct]
| stats max(*_ct) AS *_ct
| eval redundant_events_ct=total_ct-unique_ct
| foreach *_ct [| eval <<MATCHSTR>>_per=ROUND((<<FIELD>>/total_ct)*100, 2)." %"]
| table unique_ct unique_per events_with_dupe_ct events_with_dupe_per redundant_events_ct redundant_events_per total_ct
  •  

 

0 Karma
Get Updates on the Splunk Community!

Observability Newsletter Highlights | March 2023

 March 2023 | Check out the latest and greatestSplunk APM's New Tag Filter ExperienceSplunk APM has updated ...

Security Newsletter Updates | March 2023

 March 2023 | Check out the latest and greatestUnify Your Security Operations with Splunk Mission Control The ...

Platform Newsletter Highlights | March 2023

 March 2023 | Check out the latest and greatestIntroducing Splunk Edge Processor, simplified data ...