topic Re: Percentile Implementation in Splunk Search

Percentile Implementation

sohrab — Mon, 02 Apr 2012 07:26:12 GMT

I am wondering what percentile implementation does Splunk use (used by stats, etc.). It does not always return the same results as Excel's or what I calculate manually (may be interpolated).

Is it the function from scipy.stats? Or it is a custom function? Is it possible to get the formula if it is custom?

Re: Percentile Implementation

steveyz — Mon, 02 Apr 2012 23:20:25 GMT

If there are less than 1000 distinct values, the percentiles use the nearest rank algorithm (see http://en.wikipedia.org/wiki/Percentile#Nearest_rank). Excel uses the NIST interpolated algorithm, which basically means you can get a value for a percentile that does not exist in the actual data, which is not possible for the nearest rank approach. You can ask splunk to use the excel method instead via a limits.conf setting [stats] perc_method=interpolated (vs 'nearest-rank'). See the limits.conf.spec entry for more detailed info.

If there are more than 1000 distinct values for the field, the percentiles are approximated using a custom radix-tree digest based algorithm that is much faster and uses much less (a constant amount) memory than an exact computation (which uses memory in linear relation to the number of distinct values). By default this approproach limits the approximation error to < 1% of rank error. That means if you ask for e.g. 95th percentile, the number you get back is between the 94th and 96th percentile.

You always get the exact percentiles even for more than 1000 distinct values by using 'exactperc' instead of 'perc'

Re: Percentile Implementation

sohrab — Tue, 03 Apr 2012 00:13:03 GMT

Thanks for the speedy response.

Re: Percentile Implementation

Ellen — Tue, 29 May 2012 17:12:15 GMT

For additional reference, in 4.3.2 you can find further details in the following files.

$SPLUNK_HOME/etc/system/default/searchbnf.conf

[stats-perc]
syntax = (perc|p|exactperc|upperperc)
simplesyntax = perc
description = The n-th percentile value of this field. perc, p, and upperperc give approximate values for the integer percentile requested. The approximation algorithm we use provides a strict bound of the actual value at for any percentile. perc and p return a single number that represents the lower end of that range while upperperc gives the approximate upper bound. exactperc provides the exact value, but will be very expensive for high cardinality fields.

$SPLUNK_HOME/etc/system/README/limits.conf.spec

perc_method = nearest-rank|interpolated
* Which method to use for computing percentiles (and medians=50 percentile).
* nearest-rank picks the number with 0-based rank R = floor((percentile/100)count)
* interpolated means given F = (percentile/100)(count-1), pick ranks R1 = floor(F) and R2 = ceiling(F). Answer = (R2 * (F - R1)) + (R1 * (1 - (F - R1)))
* See wikipedia percentile entries on nearest rank and "alternative methods"
* Defaults to interpolated

Re: Percentile Implementation

sundarrajan — Thu, 09 Feb 2017 13:46:08 GMT

@steveyz,
Dear Steve, is it possible for us to get a sneak peak into rdigist algorithm or any "custom built radix tree digist algorithm" for knowledge purpose. In 6.4 we could see by-default splunk takes "closerank" algorithm over "interpolated".