Splunk Search

What are some solutions for high cardinality field reporting?

Champion

We have high cardinality data -- virtually every event is unique except for a small percentage of cases that we care about. So we're finding that we have count the unique ids, track them somehow in order to find the duplicates. Its just not feasible in Splunk when we have millions of events per minute.

Example:

search | stats count by unique_id | where count>1

(Millions of events per minute, results in a few hundred events where count>1). Summary indexing is not really a solution here since the unique_id could cross time/minute boundary.

0 Karma

Contributor

|fields unique_id
|stats list( unique_id) AS id
| where mvfilter(mvcount(id)>1) != NULL

0 Karma

Revered Legend

May be setup indexed time field extraction for your unique_id field so you could use tstats with it.

State of Splunk Careers

Access the Splunk Careers Report to see real data that shows how Splunk mastery increases your value and job satisfaction.

Find out what your skills are worth!