Splunk Search

Percentile Implementation

Explorer

Hi

I am wondering what percentile implementation does Splunk use (used by stats, etc.). It does not always return the same results as Excel's or what I calculate manually (may be interpolated).

Is it the function from scipy.stats? Or it is a custom function? Is it possible to get the formula if it is custom?

Tags (2)
1 Solution

Splunk Employee
Splunk Employee

If there are less than 1000 distinct values, the percentiles use the nearest rank algorithm (see http://en.wikipedia.org/wiki/Percentile#Nearest_rank). Excel uses the NIST interpolated algorithm, which basically means you can get a value for a percentile that does not exist in the actual data, which is not possible for the nearest rank approach. You can ask splunk to use the excel method instead via a limits.conf setting [stats] perc_method=interpolated (vs 'nearest-rank'). See the limits.conf.spec entry for more detailed info.

If there are more than 1000 distinct values for the field, the percentiles are approximated using a custom radix-tree digest based algorithm that is much faster and uses much less (a constant amount) memory than an exact computation (which uses memory in linear relation to the number of distinct values). By default this approproach limits the approximation error to < 1% of rank error. That means if you ask for e.g. 95th percentile, the number you get back is between the 94th and 96th percentile.

You always get the exact percentiles even for more than 1000 distinct values by using 'exactperc' instead of 'perc'

View solution in original post

Splunk Employee
Splunk Employee

For additional reference, in 4.3.2 you can find further details in the following files.

$SPLUNK_HOME/etc/system/default/searchbnf.conf

[stats-perc]
syntax = (perc|p|exactperc|upperperc)
simplesyntax = perc
description = The n-th percentile value of this field. perc, p, and upperperc give approximate values for the integer percentile requested. The approximation algorithm we use provides a strict bound of the actual value at for any percentile. perc and p return a single number that represents the lower end of that range while upperperc gives the approximate upper bound. exactperc provides the exact value, but will be very expensive for high cardinality fields.

$SPLUNK_HOME/etc/system/README/limits.conf.spec

perc_method = nearest-rank|interpolated
* Which method to use for computing percentiles (and medians=50 percentile).
* nearest-rank picks the number with 0-based rank R = floor((percentile/100)count)
* interpolated means given F = (percentile/100)
(count-1), pick ranks R1 = floor(F) and R2 = ceiling(F). Answer = (R2 * (F - R1)) + (R1 * (1 - (F - R1)))
* See wikipedia percentile entries on nearest rank and "alternative methods"
* Defaults to interpolated

0 Karma

Splunk Employee
Splunk Employee

If there are less than 1000 distinct values, the percentiles use the nearest rank algorithm (see http://en.wikipedia.org/wiki/Percentile#Nearest_rank). Excel uses the NIST interpolated algorithm, which basically means you can get a value for a percentile that does not exist in the actual data, which is not possible for the nearest rank approach. You can ask splunk to use the excel method instead via a limits.conf setting [stats] perc_method=interpolated (vs 'nearest-rank'). See the limits.conf.spec entry for more detailed info.

If there are more than 1000 distinct values for the field, the percentiles are approximated using a custom radix-tree digest based algorithm that is much faster and uses much less (a constant amount) memory than an exact computation (which uses memory in linear relation to the number of distinct values). By default this approproach limits the approximation error to < 1% of rank error. That means if you ask for e.g. 95th percentile, the number you get back is between the 94th and 96th percentile.

You always get the exact percentiles even for more than 1000 distinct values by using 'exactperc' instead of 'perc'

View solution in original post

Explorer

Thanks for the speedy response.

0 Karma

Path Finder

@steveyz,
Dear Steve, is it possible for us to get a sneak peak into rdigist algorithm or any "custom built radix tree digist algorithm" for knowledge purpose. In 6.4 we could see by-default splunk takes "closerank" algorithm over "interpolated".

0 Karma
State of Splunk Careers

Access the Splunk Careers Report to see real data that shows how Splunk mastery increases your value and job satisfaction.

Find out what your skills are worth!