Splunk Search

Stdev has a weird behaviour

Mahieu
Communicator

Hi everyone,

I'm seeing strange results using stdev.
I'm using the following command :

sourcetype=whatever | stats avg(timetaken) as average, stdev(timetaken) as deviation by uri | where deviation > average

And I have results that are actually showing a standard deviation that is superior to the average ... which is not mathematically possible according to me unless we have negative "timetaken" values (and that's not the case).

I tried stdevp too but it's not better.

Has anyone got a clue ?

Thanks in advance for your help.

Regards,

Mat

Tags (2)
1 Solution

jtrucks
Splunk Employee
Splunk Employee

This is entirely mathematically possible if you have a large range of numbers, especially when not evenly distributed.

From http://staff.argyll.epsb.ca/jreed/math30p/statistics/standardDeviation.htm the example is:

The standard deviation can be larger
than the mean. The range of numbers
has to be large for this to occur. One
such sample is {0, 0, 1, 15, 20},
where [the mean] = 7.2 and [the standard deviation] = 8.6.

--
Jesse Trucks
Minister of Magic

View solution in original post

prelert
Path Finder

Don't forget that "standard deviation" assumes that the values are distributed in a symmetrical, Gaussian ("Normal" or "Bell") curve. As jtruck's example shows, certain data can exist that doesn't really conform to this Gaussian curve and thus the standard deviation function becomes the wrong tool to analyze the behavior of the data.

VMWare once did a nice whitepaper (http://www.vmware.com/files/pdf/vcenter/VMware-vCenter-Operations-How-Normal-Is-Your-Data-WP-EN.pdf) that explains just now non-"Normal" machine data can be.

0 Karma

jtrucks
Splunk Employee
Splunk Employee

This is entirely mathematically possible if you have a large range of numbers, especially when not evenly distributed.

From http://staff.argyll.epsb.ca/jreed/math30p/statistics/standardDeviation.htm the example is:

The standard deviation can be larger
than the mean. The range of numbers
has to be large for this to occur. One
such sample is {0, 0, 1, 15, 20},
where [the mean] = 7.2 and [the standard deviation] = 8.6.

--
Jesse Trucks
Minister of Magic

Mahieu
Communicator

Alright, thanks for that.
I'll go back to the Math class then 😉

0 Karma
Get Updates on the Splunk Community!

Adoption of RUM and APM at Splunk

    Unleash the power of Splunk Observability   Watch Now In this can't miss Tech Talk! The Splunk Growth ...

March Community Office Hours Security Series Uncovered!

Hello Splunk Community! In March, Splunk Community Office Hours spotlighted our fabulous Splunk Threat ...

Stay Connected: Your Guide to April Tech Talks, Office Hours, and Webinars!

Take a look below to explore our upcoming Community Office Hours, Tech Talks, and Webinars in April. This post ...