How to get a daily count of distinct users over a large time range without re-running from beginning of time

[e_sherlock]

I have a simple "| stats dc()" command to get a cumulative sum of distinct users; however, I don't want to have to run this query from the beginning of time every day it runs for performance reasons.

These couple routes came to mind, but don't seem best...

1. Summary indexes can write out the distinct users per day, but seems like couldn't compute distinct over the whole set of days.
2. Write the distinct values to lookup and only adding the new values each day -- a kinda hacky incremental approach.

Any ideas on the optimal solution?

[gkanapathy · Accepted solution]

The best way would be to use sistats and a summary, or alternatively, use report acceleration. With report acceleration, just set up your search and tell Splunk to accelerate, and that should do it.

If you summarize yourself, then on a daily basis (and you can backfill later), you run a;

... | sistats dc(user) by x,y,z 

and store that to a summary. Then to get your counts:

index=my_summary_index name=my_summary_job | stats dc(user)

(or ... | stats dc(user) by x,y,z or ... | stats dc(user) by x,y).

the sistats command will have saved the right data, and the stats command will know how to handle what sistats did. Yes, it is doing slightly clever things under the hood.