Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.

Showing results for

Splunk Search

Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.

Showing results for

- Community
- :
- Splunk Answers
- :
- Using Splunk
- :
- Splunk Search
- :
- Percentile Implementation

- Subscribe to RSS Feed
- Mark Topic as New
- Mark Topic as Read
- Float this Topic for Current User
- Bookmark Topic
- Subscribe to Topic
- Mute Topic
- Printer Friendly Page

- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Email to a Friend
- Report Inappropriate Content

sohrab

Explorer

04-02-2012
12:26 AM

Hi

I am wondering what percentile implementation does Splunk use (used by stats, etc.). It does not always return the same results as Excel's or what I calculate manually (may be interpolated).

Is it the function from scipy.stats? Or it is a custom function? Is it possible to get the formula if it is custom?

1 Solution

- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Email to a Friend
- Report Inappropriate Content

steveyz

Splunk Employee

04-02-2012
04:20 PM

If there are less than 1000 distinct values, the percentiles use the nearest rank algorithm (see http://en.wikipedia.org/wiki/Percentile#Nearest_rank). Excel uses the NIST interpolated algorithm, which basically means you can get a value for a percentile that does not exist in the actual data, which is not possible for the nearest rank approach. You can ask splunk to use the excel method instead via a limits.conf setting [stats] perc_method=interpolated (vs 'nearest-rank'). See the limits.conf.spec entry for more detailed info.

If there are more than 1000 distinct values for the field, the percentiles are approximated using a custom radix-tree digest based algorithm that is much faster and uses much less (a constant amount) memory than an exact computation (which uses memory in linear relation to the number of distinct values). By default this approproach limits the approximation error to < 1% of rank error. That means if you ask for e.g. 95th percentile, the number you get back is between the 94th and 96th percentile.

You always get the exact percentiles even for more than 1000 distinct values by using 'exactperc' instead of 'perc'

- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Email to a Friend
- Report Inappropriate Content

Ellen

Splunk Employee

05-29-2012
10:12 AM

For additional reference, in 4.3.2 you can find further details in the following files.

$SPLUNK_HOME/etc/system/default/searchbnf.conf

[stats-perc]

syntax = (perc|p|exactperc|upperperc)

simplesyntax = perc

description = The n-th percentile value of this field. perc

$SPLUNK_HOME/etc/system/README/limits.conf.spec

perc_method = nearest-rank|interpolated

* Which method to use for computing percentiles (and medians=50 percentile).

* nearest-rank picks the number with 0-based rank R = floor((percentile/100)*count)
* interpolated means given F = (percentile/100)*(count-1), pick ranks R1 = floor(F) and R2 = ceiling(F). Answer = (R2 * (F - R1)) + (R1 * (1 - (F - R1)))

* See wikipedia percentile entries on nearest rank and "alternative methods"

* Defaults to interpolated

- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Email to a Friend
- Report Inappropriate Content

steveyz

Splunk Employee

04-02-2012
04:20 PM

If there are less than 1000 distinct values, the percentiles use the nearest rank algorithm (see http://en.wikipedia.org/wiki/Percentile#Nearest_rank). Excel uses the NIST interpolated algorithm, which basically means you can get a value for a percentile that does not exist in the actual data, which is not possible for the nearest rank approach. You can ask splunk to use the excel method instead via a limits.conf setting [stats] perc_method=interpolated (vs 'nearest-rank'). See the limits.conf.spec entry for more detailed info.

If there are more than 1000 distinct values for the field, the percentiles are approximated using a custom radix-tree digest based algorithm that is much faster and uses much less (a constant amount) memory than an exact computation (which uses memory in linear relation to the number of distinct values). By default this approproach limits the approximation error to < 1% of rank error. That means if you ask for e.g. 95th percentile, the number you get back is between the 94th and 96th percentile.

You always get the exact percentiles even for more than 1000 distinct values by using 'exactperc' instead of 'perc'

- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Email to a Friend
- Report Inappropriate Content

sohrab

Explorer

04-02-2012
05:13 PM

Thanks for the speedy response.

- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Email to a Friend
- Report Inappropriate Content

sundarrajan

Path Finder

02-09-2017
05:46 AM

Dear Steve, is it possible for us to get a sneak peak into rdigist algorithm or any "custom built radix tree digist algorithm" for knowledge purpose. In 6.4 we could see by-default splunk takes "closerank" algorithm over "interpolated".

State of Splunk Careers

Find out what your skills are worth!

Read the report >