Hi
I am wondering what percentile implementation does Splunk use (used by stats, etc.). It does not always return the same results as Excel's or what I calculate manually (may be interpolated).
Is it the function from scipy.stats? Or it is a custom function? Is it possible to get the formula if it is custom?
If there are less than 1000 distinct values, the percentiles use the nearest rank algorithm (see http://en.wikipedia.org/wiki/Percentile#Nearest_rank). Excel uses the NIST interpolated algorithm, which basically means you can get a value for a percentile that does not exist in the actual data, which is not possible for the nearest rank approach. You can ask splunk to use the excel method instead via a limits.conf setting [stats] perc_method=interpolated (vs 'nearest-rank'). See the limits.conf.spec entry for more detailed info.
If there are more than 1000 distinct values for the field, the percentiles are approximated using a custom radix-tree digest based algorithm that is much faster and uses much less (a constant amount) memory than an exact computation (which uses memory in linear relation to the number of distinct values). By default this approproach limits the approximation error to < 1% of rank error. That means if you ask for e.g. 95th percentile, the number you get back is between the 94th and 96th percentile.
You always get the exact percentiles even for more than 1000 distinct values by using 'exactperc' instead of 'perc'
For additional reference, in 4.3.2 you can find further details in the following files.
$SPLUNK_HOME/etc/system/default/searchbnf.conf
[stats-perc]
syntax = (perc|p|exactperc|upperperc)
simplesyntax = perc
description = The n-th percentile value of this field. perc
$SPLUNK_HOME/etc/system/README/limits.conf.spec
perc_method = nearest-rank|interpolated
* Which method to use for computing percentiles (and medians=50 percentile).
* nearest-rank picks the number with 0-based rank R = floor((percentile/100)count)
* interpolated means given F = (percentile/100)(count-1), pick ranks R1 = floor(F) and R2 = ceiling(F). Answer = (R2 * (F - R1)) + (R1 * (1 - (F - R1)))
* See wikipedia percentile entries on nearest rank and "alternative methods"
* Defaults to interpolated
If there are less than 1000 distinct values, the percentiles use the nearest rank algorithm (see http://en.wikipedia.org/wiki/Percentile#Nearest_rank). Excel uses the NIST interpolated algorithm, which basically means you can get a value for a percentile that does not exist in the actual data, which is not possible for the nearest rank approach. You can ask splunk to use the excel method instead via a limits.conf setting [stats] perc_method=interpolated (vs 'nearest-rank'). See the limits.conf.spec entry for more detailed info.
If there are more than 1000 distinct values for the field, the percentiles are approximated using a custom radix-tree digest based algorithm that is much faster and uses much less (a constant amount) memory than an exact computation (which uses memory in linear relation to the number of distinct values). By default this approproach limits the approximation error to < 1% of rank error. That means if you ask for e.g. 95th percentile, the number you get back is between the 94th and 96th percentile.
You always get the exact percentiles even for more than 1000 distinct values by using 'exactperc' instead of 'perc'
Thanks for the speedy response.
@steveyz,
Dear Steve, is it possible for us to get a sneak peak into rdigist algorithm or any "custom built radix tree digist algorithm" for knowledge purpose. In 6.4 we could see by-default splunk takes "closerank" algorithm over "interpolated".