Splunk Search

What are some solutions for high cardinality field reporting?


We have high cardinality data -- virtually every event is unique except for a small percentage of cases that we care about. So we're finding that we have count the unique ids, track them somehow in order to find the duplicates. Its just not feasible in Splunk when we have millions of events per minute.


search | stats count by unique_id | where count>1

(Millions of events per minute, results in a few hundred events where count>1). Summary indexing is not really a solution here since the unique_id could cross time/minute boundary.

0 Karma


|fields unique_id
|stats list( unique_id) AS id
| where mvfilter(mvcount(id)>1) != NULL

0 Karma

Revered Legend

May be setup indexed time field extraction for your unique_id field so you could use tstats with it.

Don’t Miss Global Splunk
User Groups Week!

Free LIVE events worldwide 2/8-2/12
Connect, learn, and collect rad prizes
and swag!