How to create a search to count Millions of Records?

[genesiusj]

Hello,

We have a lookup/kvstore containing over 3.M records*. We need to count the number of times each value is found over all of the records.

Ex: Count the occurrence of the same LAST_NAME

Field Name: LAST_NAME
Values: JONES, SMITH, DAVIS, GARCIA
Counters Values: 12, 34, 16, 23

This is just one of several different counters: BIRTH_YEAR, CITY, STATE, etc.

Because of the limits within Splunk, this code would result in blanks and inaccurate counts.

| eventstats count(ID) as count_same_city by CITY 

Any suggestions?

* The number of records increases by 10K every week.

Thanks in advance, and God bless,
Genesius

[ITWhisperer]

Create a summary index which holds the daily or weekly aggregated statistics, then sum the various counts over a longer period as appropriate.