Splunk Search

How to get a daily count of distinct users over a large time range without re-running from beginning of time

e_sherlock
Explorer

I have a simple "| stats dc()" command to get a cumulative sum of distinct users; however, I don't want to have to run this query from the beginning of time every day it runs for performance reasons.

These couple routes came to mind, but don't seem best...

  1. Summary indexes can write out the distinct users per day, but seems like couldn't compute distinct over the whole set of days.
  2. Write the distinct values to lookup and only adding the new values each day -- a kinda hacky incremental approach.

Any ideas on the optimal solution?

Tags (2)
0 Karma
1 Solution

gkanapathy
Splunk Employee
Splunk Employee

The best way would be to use sistats and a summary, or alternatively, use report acceleration. With report acceleration, just set up your search and tell Splunk to accelerate, and that should do it.

If you summarize yourself, then on a daily basis (and you can backfill later), you run a;

... | sistats dc(user) by x,y,z 

and store that to a summary. Then to get your counts:

index=my_summary_index name=my_summary_job | stats dc(user)

(or ... | stats dc(user) by x,y,z or ... | stats dc(user) by x,y).

the sistats command will have saved the right data, and the stats command will know how to handle what sistats did. Yes, it is doing slightly clever things under the hood.

View solution in original post

gkanapathy
Splunk Employee
Splunk Employee

The best way would be to use sistats and a summary, or alternatively, use report acceleration. With report acceleration, just set up your search and tell Splunk to accelerate, and that should do it.

If you summarize yourself, then on a daily basis (and you can backfill later), you run a;

... | sistats dc(user) by x,y,z 

and store that to a summary. Then to get your counts:

index=my_summary_index name=my_summary_job | stats dc(user)

(or ... | stats dc(user) by x,y,z or ... | stats dc(user) by x,y).

the sistats command will have saved the right data, and the stats command will know how to handle what sistats did. Yes, it is doing slightly clever things under the hood.

*NEW* Splunk Love Promo!
Snag a $25 Visa Gift Card for Giving Your Review!

It's another Splunk Love Special! For a limited time, you can review one of our select Splunk products through Gartner Peer Insights and receive a $25 Visa gift card!

Review:





Or Learn More in Our Blog >>